Computer vision system

ABSTRACT

The field of the invention relates to computer vision systems and methods providing real time data analytics on detected people or objects in the home environment or other environments. It is based on an embedded engine that analyses an image from a raw sensor and virtualised the image into a digital representation enabling a digital understanding of the environment while guarantying privacy. It comprises multiple image processing blocks and embedded firmware.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/546,794, filed on Jul. 27, 2017, which claims the priority ofPCT/GB2016/050201, filed on Jan. 29, 2016, which claims the benefit ofpriority to Great Britain Application No. GB 1501510.0, filed on Jan.29, 2015, the entire contents of each of which are fully incorporatedherein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The field of the invention relates to computer vision systems andmethods providing real time data analytics on detected people or objectsin the home environment or other environments.

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

2. Technical Background

The last few years have seen enormous advances in computer visionresearch and many teams are now attempting to deploy applications basedon so-called intelligent video analysis. These usually fall into twocamps: those doing very simple video analysis inside cameras, and thosedoing sophisticated video analysis on servers. However, the simple videoanalysis is too simple and unpredictable to be of much value inapplications, while the sophisticated analysis is often completelyun-scalable and uneconomical.

The reason for the latter is that state-of-the-art vision algorithms areunsuited for processing high-resolution video on CPUs, GPUs, and evendedicated vision processors: the costs of transmitting, storing andespecially processing video are huge and scale linearly with the numberof users, cameras and resolution.

Other systems have failed to provide an accurate and predictablesolution for image and video processing implemented in a low-cost andlow power sensor. Aspects of this invention address these failures.

Home automation/Smart home relates to the local and/or remote control ofvarious devices or systems that are present in homes. Today's smart homeservices include a wide range of services, such as for example,automated door locks, energy monitoring devices, smart lighting systems,entertainment systems, vehicle detection systems and so on.

A Smart Home's connected devices must be able to monitor, contextualizeand predict human behaviour as well as guarantee privacy. Smart Homesneed to place the person at the heart of and in control of the home.Smart Home systems need to be functionally rich, accurate, predictableand able to interpret human behaviour. Current available solutions onthe market are unable to deliver.

3. Discussion of Related Art

Various techniques providing data analytics on detected people orobjects in an home environment are available. Examples are:

-   -   PIR sensors register only motion within their field of view.        They fail when people are not moving, and give no information on        the number of people present, their locations, their poses or        their identities. They are triggered equally by any kind of        movement.    -   Cloud cameras require transmission of full-frame video to a        remote server and subsequent video analytics. Due to the very        high computational requirements of the necessary algorithms,        such systems are not real-time and are un-scalable as the        storage and computational costs grow linearly with the number of        sensors. In addition, they provide no possibility of        guaranteeing privacy.    -   Gesture systems (such as Microsoft Kinect™) provide rich        information but are close-range only, work reliably in a        relatively narrow range of lighting conditions, and are        relatively bulky and expensive.

SUMMARY OF THE INVENTION

A first aspect is a computer-vision system or engine that (a) generatesfrom a pixel stream a digital representation of a person or other objectand (b) determines attributes or characteristics of the person or objectfrom that digital representation and (c) enables one or more networkeddevices or sensors to be controlled.

Optional features include the following, each of which can be combinedwith any other optional feature. In the following, the term ‘engine’also includes the term ‘system’.

-   -   The engine outputs a real-time stream of metadata from the pixel        stream, the metadata describing the instantaneous attributes or        characteristics of each object in the scene it has been trained        to search for.    -   The engine directly processes raw image sensor data or video        data in the form of RGB, YUV or other encoding formats.    -   The engine is an ASIC based product embedded in a device.    -   The engine sends or uses those attributes or characteristics to        enable one or more networked devices or sensors to be        controlled.    -   The engine can detect multiple people in a scene and        continuously track or detect one or more of their: trajectory,        pose, gesture, identity.    -   The engine can infer or describe a person's behaviour or intent        by analysing one or more of the trajectory, pose, gesture,        identity of that person.    -   The engine performs real-time virtualisation of the scene,        extracting objects from the scene and grouping their virtualised        representations together.    -   The engine applies feature extraction and classification to find        objects of known characteristics in each video frame or applies        a convolutional or recurrent neural network or another object        detection algorithm to do so.    -   The engine detects people by extracting independent        characteristics including one or more of the following: the        head, head & shoulders, hands and full body, each in different        orientations, to enable an individual's head orientation,        shoulder orientation and full body orientation to be        independently evaluated for reliable people tracking.    -   The engine continuously monitors the motion of individuals in        the scene and predicts their next location to enable reliable        tracking even when the subject is temporarily lost or passes        behind another object.    -   The engine contextualizes individual local representations to        construct a global representation of each person as they move        through an environment of multiple sensors in multiple        locations.    -   The engine uses data from multiple sensors, each capturing        different parts of an environment, to track and show an object        moving through that environment and to form a global        representation that is not limited to the object when imaged        from a single sensor.    -   Approximate location of the object in 3D is reconstructed using        depth/distance estimation to assist accuracy of tracking and        construction of the global representation from multiple sensors.    -   The engine operates as an interface to enable control of        multiple, networked computer-enabled sensors and devices in the        smart home or office.    -   The digital representation conforms to an API.    -   The digital representation includes feature vectors that define        the appearance of a generalized person.    -   The digital representation is used to display a person as a        standardised shape.    -   The digital representation is used to display a person as a        symbolic or simplified representation of a person.    -   The symbolic or simplified representation is a flat or        2-dimensional shape including head, body, arms and legs.    -   The symbolic or simplified representations of different people        are distinguished using different colours.    -   The symbolic or simplified representation is an avatar.    -   The digital representation includes feature vectors that define        the appearance of a specific person.    -   The digital representation of a person is used to analyse, or        enable the analysis of one or more of trajectory, pose, gesture        and identity of that person and smart home devices can respond        to and predict the person's intent and/or needs based on that        analysis.    -   The digital representation is not an image and does not enable        an image of a person to be created from which that person can be        recognised.    -   The engine does not output continuous or streaming video but        instead metadata that defines various attributes of individual        persons.    -   The engine outputs continuous or streaming video and also        metadata that defines various attributes of individual persons.    -   The characteristics or attributes include one or more of        trajectory, pose, gesture, identity.    -   The characteristics or attributes include each of trajectory,        pose, gesture, and identity.    -   The engine works with standard images sensors working with        chip-level systems that generate real-time data that enables a        digital representation of people or other objects to be created.    -   The engine works with IP cameras to form a real-time metadata        stream to accompany the output video stream providing an index        of video content frame by frame.    -   The engine works with smart sensors that use visual information,        but never form imagery or video at a hardware level.    -   System builds a virtualized digital representation of each        individual in the home, comprising each individual's: Trajectory        around the home, including for example the actions of standing        and sitting; Pose, for example in which direction the person is        facing, and/or in which direction they are looking; Gesture, for        example motions made by the person's hands; and Identity, namely        the ability to differentiate between people and assign a unique        identity (e.g. name) to each person.    -   System understands a wide range of behaviours from the set:        counting the number of people in the room, understanding        people's pose, identifying persons using facial recognition        data, determining where people are moving from/to, extracting        specific gestures by an identified individual.    -   Data rate of the data sent from a computer-vision system is        throttled up or based on event-triggering.    -   Multiple computer-vision systems send their data to a hub that        stores and analyses that data and enables a digital        representation of a person to be constructed from        computer-vision systems with both shared and differing fields of        view, tracking that person and also recognizing that person.    -   The hub exposes an open, person-level digital representation        API, enabling various appliances to use and to be controlled in        dependence on the data encoded in the API.    -   Digital representation is created locally at a computer-vision        system, or at a hub, or in the cloud, or distributed across        computer-vision systems and one or more hubs and the cloud.    -   Digital representation is a ‘track record’ that uses the        reformatting of real-time metadata into a per-object (e.g.        per-person) record of their trajectory, pose (and, possibly,        identity) of that object.    -   The Track Records are stored in a MySQL-type database,        correlated with a video database.    -   Digital representation includes an estimate or measurement of        depth or distance from the sensor of a person or object or part        of the environment.    -   Depth sensing uses a calibration object of approximately known        size,    -   Depth sensing uses stereoscopic cameras or structured light.    -   Digital representation includes facial recognition data.    -   Sensor metadata is fed into a hub, gateway or controller that        pushes events to smart devices in a network as specific        commands, and differentiates the events created on a per service        basis to allow each service to receive different data that is        relevant to their service from the group of sensors as a single        intelligent sensor.    -   The event streams are sent to cloud analytics apps such as for        example cloud-based data monitoring, data gathering or learning        service.    -   An event subscription service, to which a system controller        subscribes, receives event notifications and data from the        devices or sensors.    -   A virtual output queued event switch is used so that events        being pushed to the control system can be differentiated by a        class of service marker.    -   System generates event objects from a collection of individual        sensor inputs in which each event object also contains        subscriber information and class of service.    -   The event objects are coded in JSON format so that they can be        directly used in Javascript-based software on Browser User        Interfaces (BUIs) and web servers, or easily interpreted by        standard server side programming languages or server Application        Programming Interfaces (APIs).    -   System queues the generated events and switches them into an        output channel based on destination and class of service using a        virtual output queuing system.    -   The digital representation relates to other items selected from        the list: animals, pets, inanimate objects, dynamic or moving        objects like cars.    -   Control is implemented using gesture recognition (e.g. wave at a        computer-vision sensor to turn it off; wave at a light switch to        turn the lights up or down).    -   Control is implemented using movement detection (e.g. approach a        room and its lights turn on; approach the sofa and the TV turns        on).    -   Voice-controlled system can be enhanced since voice commands can        be dis-ambiguated—e.g. reliably identified as commands and not        background noise since a user can be seen to be looking at the        microphone or other sensor or object to be controlled when        giving the command and also voice controlled system can be set        to monitor audio only when user is seen to be looking at the        microphone/object to enhance privacy.    -   System or engine can be localised in a camera or other device        including a sensor, or in a hub or gateway connected to that        device, or in a remote server, or distributed across any        permutation of these.    -   System or engine is localised in one or more of the        following: (a) an edge layer that processes raw sensor data; (b)        an aggregation layer that provides high level analytics by        aggregating and processing data from the edge layer in the        temporal and spatial domains; (c) a service layer that handles        all connectivity to one or more system controllers and to the        end customers for configuration of their home systems and the        collection and analysis of the data produced.

Other aspects of the invention (which may also include any of theoptional features listed above) include the following:

A computer-vision system or engine that (a) generates from video adigital representation or virtualisation of a person or other object and(b) determines attributes or characteristics of the person or objectfrom that digital representation and (c) enables one or more networkeddevices or sensors to be controlled.

A sensor that includes an embedded computer-vision engine that (a)generates from a pixel stream a digital representation of a person orother object and (b) determines attributes or characteristics of theperson or object from that digital representation.

An appliance that includes a sensor that in turn includes an embeddedcomputer-vision engine that (a) generates from a pixel stream a digitalrepresentation of a person or other object and (b) determines attributesor characteristics of the person or object from that digitalrepresentation and (c) enables one or more networked devices or sensorsto be controlled.

A smart home or office system (and/or other physical or logicalenvironment) including one or more computer-vision systems as definedabove and one or more sensors as defined above.

A networked system including multiple computer-vision systems as definedabove.

Chip-level hardware or firmware that enables a computer vision engine tooperate as defined above.

A method of controlling multiple, networked computer-enabled devicesusing the computer-vision systems as defined above.

Methods or systems as defined above, where one or more computer visionengines, as defined above, are embedded in one of the followingproducts:

-   -   Camera    -   Cloud camera    -   Smart Door Bell    -   Light Switch    -   Garage entry system    -   Non-camera sensor based    -   Fire alarm sensor or alarm    -   TV    -   Thermostat    -   Coffee machine    -   Light bulb    -   Music steaming device    -   Fridge    -   Oven    -   Microwave cooker    -   Washing machine    -   Any smart device    -   Any wearable computing device    -   Smartphone    -   Tablet    -   Any portable computing device

Another aspect is a software architecture or system for a smart home orsmart office, the architecture including

-   (a) an edge layer that processes sensor data;-   (b) an aggregation layer that provides high level analytics by    aggregating and processing data from the edge layer in the temporal    and spatial domains;-   (c) a service layer that handles all connectivity to one or more    system controllers and to the end customers for configuration of    their home systems and the collection and analysis of the data    produced.

Optional features include the following, each of which can be combinedwith any other optional feature, as well as any of the aspects andoptional features listed above.

-   -   the edge layer processes raw sensor data or video data at an        ASIC embedded in a sensor or at a gateway/hub;    -   The edge layer includes a computer-vision system or engine        that (a) generates from a pixel stream a digital representation        of a person or other object and (b) determines attributes or        characteristics of the person or object from that digital        representation and (c) enables one or more networked devices or        sensors to be controlled.    -   The edge layer can detect multiple people in a scene and        continuously track or detect one or more of their: trajectory,        pose, gesture, identity.    -   The edge layer can infer or describe a person's behaviour or        intent by analysing one or more of the trajectory, pose,        gesture, identity of that person.    -   The computer vision system is as defined above.    -   The computer vision system uses stereoscopic cameras or        structured light.    -   The system continuously analyses each person it is sensing and        interprets certain behaviours as events.    -   The edge layer pushes real-time metadata from the raw sensor        data to the aggregation layer.    -   The aggregation layer takes the metadata produced by the edge        layer and analyses it further, combining multiple sources of        data together to create events as functions of time.    -   The aggregation layer interprets a set of rules for the creation        of events.    -   The aggregation layer prepares the events for delivery as a        service, which includes scheduling algorithms that drive a        multi-class of service event switch before passing the event        data through to the service layer.    -   The service layer allows the system to interact with real-time        control systems that subscribe for an event service that is        packaged, delivered and monitored by the service layer.    -   All 3 layers of the architecture or system are contained within        a gateway or hub device, to which cameras or other sensors are        connected, and a portion of the service layer is in the cloud.    -   The gateway or hub component of the edge layer is used to        centralise some management components of the architecture rather        than replicate them across all of the cameras/ sensors        themselves.    -   Cameras or other sensors include some of the edge layer, and        these elements of the edge layer output real-time metadata; all        3 layers of the architecture are contained within a gateway or        hub device, to which the cameras or other sensors are connected,        and a portion of the service layer is in the cloud.    -   Cameras or other sensors include some of the edge layer, and        these elements of the edge layer output real-time metadata; all        3 layers of the architecture are in the cloud.

Implementations can be deployed in any of the following markets:

-   -   Smart Home    -   Smart Office    -   Smart City    -   Healthcare    -   Environment control    -   Retail/advertising    -   Home Security    -   Insurance    -   Education

Another aspect is a chip-level hardware or firmware computer visionengine that (a) generates from a pixel stream a digital representationor virtualisation of a person or other object and (b) determinesattributes or characteristics of the person or object from that digitalrepresentation.

Optional features include the following, each of which can be combinedwith any other optional feature:

-   -   The engine outputs a real-time stream of metadata from the pixel        stream, the metadata describing the instantaneous attributes or        characteristics of each object in the scene it has been trained        to search for.    -   The engine directly processes raw image sensor data or video        data in the form of RGB, YUV or other encoding formats.    -   The engine is an ASIC based product embedded in a device.    -   The engine detects multiple people in a scene and continuously        track or detect one or more of their: trajectory, pose, gesture,        identity.    -   The engine can infer or describe a person's behaviour or intent        by analysing one or more of the trajectory, pose, gesture,        identity of that person.    -   The engine performs real-time virtualisation of the scene,        extracting objects from the scene and grouping their virtualised        representations together.    -   The engine applies feature extraction and classification to find        objects of known characteristics in each video frame or applies        a convolutional or recurrent neural network or another object        detection algorithm to do so.    -   The engine detects people by extracting independent        characteristics including one or more of the following: the        head, head & shoulders, hands and full body, each in different        orientations, to enable an individual's head orientation,        shoulder orientation and full body orientation to be        independently evaluated for reliable people tracking.    -   The engine continuously monitors the motion of individuals in        the scene and predicts their next location to enable reliable        tracking even when the subject is temporarily lost or passes        behind another object.    -   The engine contextualizes individual local representations to        construct a global representation of each person as they move        through an environment of multiple sensors in multiple        locations.    -   The engine uses data from multiple sensors, each capturing        different parts of an environment, to track and show an object        moving through that environment and to form a global        representation that is not limited to the object when imaged        from a single sensor.    -   Approximate location of the object in 3D is reconstructed using        depth/distance estimation to assist accuracy of tracking and        construction of the global representation from multiple sensors.    -   The digital representation conforms to an API.    -   The digital representation includes feature vectors that define        the appearance of a generalized person.    -   The digital representation of a person is used to analyse, or        enable the analysis of one or more of trajectory, pose, gesture        and identity of that person and smart home devices can respond        to and predict the person's intent and/or needs based on that        analysis.    -   The digital representation is not an image and does not enable        an image of a person to be created from which that person can be        recognised.    -   The engine does not output continuous or streaming video but        instead metadata that defines various attributes of individual        persons.    -   The engine outputs continuous or streaming video and also        metadata that defines various attributes of individual persons.    -   The characteristics or attributes include one or more of        trajectory, pose, gesture, identity.    -   The characteristics or attributes include each of trajectory,        pose, gesture, and identity.    -   The engine works with standard images sensors working with        chip-level systems that generate real-time data that enables a        digital representation of people or other objects to be created.    -   The engine works with IP cameras to form a real-time metadata        stream to accompany the output video stream providing an index        of video content frame by frame.    -   The engine works with smart sensors that use visual information,        but never form imagery or video at a hardware level.    -   System builds a virtualized digital representation of each        individual in the home, comprising each individual's: Trajectory        around the home, including for example the actions of standing        and sitting; Pose, for example in which direction the person is        facing, and/or in which direction they are looking; Gesture, for        example motions made by the person's hands; and Identity, namely        the ability to differentiate between people and assign a unique        identity (e.g. name) to each person.    -   System understands a wide range of behaviours from the set:        counting the number of people in the room, understanding        people's pose, identifying persons using facial recognition        data, determining where people are moving from/to, extracting        specific gestures by an identified individual.    -   Data rate of the data sent from a computer-vision system is        throttled up or based on event-triggering.    -   Digital representation is a ‘track record’ that uses the        reformatting of real-time metadata into a per-object (e.g.        per-person) record of their trajectory, pose (and, possibly,        identity) of that object.    -   The Track Records are stored in a MySQL-type database,        correlated with a video database.    -   Digital representation includes an estimate or measurement of        depth or distance from the sensor of a person or object or part        of the environment.    -   Depth sensing uses a calibration object of approximately known        size.    -   Depth sensing uses stereoscopic cameras or structured light.    -   Digital representation includes facial recognition data.    -   The digital representation relates to other items selected from        the list: animals, pets, inanimate objects, dynamic or moving        objects like cars.    -   Control is implemented using gesture recognition (e.g. wave at a        computer-vision sensor to turn it off; wave at a light switch to        turn the lights up or down).    -   Control is implemented using movement detection (e.g. approach a        room and its lights turn on; approach the sofa and the TV turns        on).    -   Voice-controlled system can be enhanced since voice commands can        be dis-ambiguated—e.g. reliably identified as commands and not        background noise since a user can be seen to be looking at the        microphone or other sensor or object to be controlled when        giving the command and also voice controlled system can be set        to monitor audio only when user is seen to be looking at the        microphone/object to enhance privacy.

Aspects of the invention are implemented in platforms called ART™,ALIVE™ and AWARE™; each of these platforms uses a computational visionengine called SPIRIT™; SPIRIT is a hardware based engine in an ASIC thatmay be embedded in a sensor.

ART is a platform which creates a digital representation of a person ina home, which can be used to control a network of smart home devices. Ittypically consists of:

-   -   An embedded Spirit engine in an ASIC in one or more sensors    -   A software application residing in a home hub    -   A software application residing in the cloud

Complements to the ART approach

-   -   The smartphone remains an important device in the smart home,        but it is no longer required as the central remote control for        all devices. Instead of “an app for every device”, the        smartphone becomes a method of configuring an ART-based system        and one method of user identification/personalisation used by        ART.    -   Voice recognition alone has limited potential in the smart home,        but combined with ART's ability to detect identity and intent        (that is, attention directed towards a specific device), voice        recognition may become effective as an interface with certain        devices within the home.

ALIVE is a platform and possibly a service for delivering new userexperiences around video. It consists of one or more of the following:

-   -   Spirit engine in a SoC inside a smartphone    -   Spirit engine in a FPGA inside a server    -   ALIVE software app on a smartphone    -   ALIVE software an a server

AWARE is a platform for converting peoples' behaviour into big data. Itconsists of:

-   -   Spirit engine in a SoC inside a camera or sensor    -   And/or Spirit engine in an FPGA in a router or server    -   AWARE server containing database(s), business logic, and        interfaces to client applications

BRIEF DESCRIPTION OF THE FIGURES

Aspects of the invention will now be described, by way of example(s),with reference to the followings, in which:

FIG. 1 shows a diagram of the system architecture integrated in thecloud.

FIG. 2 illustrates a method of capturing a raw image sensor data andconverting it into a metadata stream to be further analysed andprocessed.

FIG. 3 shows a comparison between the compression from a Spirit engineas compared to standard HEVC for a raw pixel stream of 4K at 60 fps.

FIG. 4 illustrates an example of analysing a person in real-time frameby frame for two subsequent frames.

FIG. 5 illustrates an example of analysing more than one person within avideo frame in real-time.

FIG. 6 shows an example of the architecture for implementing Spiritwithin a Smart Camera and Smart Sensor.

FIG. 7 illustrates block diagram of a standard video camera as comparedto the black diagrams of an implementation of Spirit with an ART smartsensor, and an ART smart camera.

FIG. 8 shows a diagram of the ALIVE platform.

FIG. 9 illustrates the different elements of ALIVE.

FIG. 10 shows an example of ‘Ghosting’.

FIG. 11 shows a screen capture of a video where various parameterswithin the video frames are extracted in real time.

FIG. 12 shows a screen capture of a video where gestures are detected inan unconstrained environment.

FIG. 13 shows the different components in ART.

FIG. 14 illustrates examples of the different possible VirtualizedDigital Representation (VDR,=BNA) of a person.

FIG. 15 shows an ART implementation inside a particular homeenvironment.

FIG. 16 shows a home Avatar.

FIG. 17 shows a diagram illustrating the different element presentbetween a local hub and a local appliance.

FIG. 18 shows a diagram representing the systematic approach to accuracyand predictability that is deployed in ART in order to process in realtime people behaviour.

FIG. 19 illustrates the place of the heart ART in the home.

FIG. 20 shows a diagram of a light switch incorporating an ART sensor.

FIG. 21 illustrates the technical details of current IP cameras asapplied to a scalable domestic CCTV proposition.

FIG. 22 shows the possible resolution and frame rates for a CCTV option.

FIG. 23 shows the different steps that are followed by an ART securitysystem.

FIG. 24 shows an example of a screen shot of the smart ART application.

FIG. 25 is a diagram of ART 3-Layer software architecture.

FIG. 26 is a diagram of the flexible architecture option 1: Hub Device &Cloud.

FIG. 27 is a diagram the flexible architecture Option 2: Sensor, GatewayDevice & Cloud.

FIG. 28 is a diagram of the flexible architecture Option 3: Sensor &Cloud.

FIG. 29 is a diagram illustrating the separation of Data Plane, ControlPlane & Management Plane.

FIG. 30 is a diagram showing Internal & External Interfaces across threeplanes of operation.

FIG. 31 is a further example of a diagram for the Flexible architectureoption 3: Sensor & Cloud.

FIG. 32 illustrates ART deployment via a Spirit-enable SoC, a camera CPUand a hub.

FIG. 33 shows an example of the Spirit embedded architecture asimplemented in the AWARE platform.

FIG. 34 shows an example of the architecture of the AWARE cloud SDKplatform.

FIG. 35 shows the different levels of pose at varying distances that canbe assessed by AWARE.

FIG. 36 shows an example of a proposed camera setup and associatedregion of interest in which a camera has been mounted above the display.

FIG. 37 shows examples of possible AWARE implementations.

FIG. 38 shows analytics graphs calculated from gaze time insights.

FIG. 39 shows an example of deep analysis of customer behaviour within ashop environment

FIG. 40 shows an example of a screen capture of a Web GUI for smartretail.

FIG. 41 is a diagram of a Smart home architecture with hub software.

FIG. 42 is a diagram of an Internet connected smart home with more roomsand occupants shown.

FIG. 43 is a diagram of a smart home with energy control system.

FIG. 44 is a diagram of a smart home with 4 controller systems: energy,safety, security and entertainment.

FIG. 45 is a diagram of a Hub Software Programme architecture andcomponents.

FIG. 46 is a diagram of an event generator module architecture andcomponents.

FIG. 47 is a diagram of an event switch module architecture andcomponents.

FIG. 48 is a diagram of a house model using JSON notation.

FIG. 49 is an example of an event in JSON notation.

FIG. 50 is an example of an event rule in JSON notation.

FIG. 51 is a diagram with an example of an application to an enterprisesingle building.

FIG. 52 is a diagram with an example of an application to an enterprisecampus of buildings.

FIG. 53 is a diagram with an example of an application to a city block.

FIG. 54 is a diagram an example of a simultaneous application to a smarthome, enterprise campus and city block.

FIG. 55 is an example of a raw data from sensor.

FIG. 56 shows examples of sensor object, group and track.

FIG. 57 is a diagram of a rule implementation.

FIG. 58 is a diagram of rule sequencing.

FIG. 59 is a diagram with an example of scheduling.

FIG. 60 shows an image capture of a kitchen environment in which an ARTsensor has been implemented.

FIG. 61 shows an image capture of a kitchen environment in which an ARTsensor has been implemented.

FIG. 62 illustrates a comparison of the typical information preservedbetween a standard video frame and an ART video frame.

FIG. 63 shows an example of a use case with a smart door bell.

FIG. 64 shows the steps when a detected face is compared with a knownlibrary database.

FIG. 65 illustrates an even triggered by an object recognised for thesmart door bell use case.

FIG. 66 shows the different steps involved in the setup of a ART doorbell system.

FIG. 67 shows a use case example of daily activity monitoring.

FIG. 68 shows an ART system demonstration of a real time scenario takingplace inside the home.

FIG. 69 illustrates an implementation of ART inside a vehicle.

FIG. 70 illustrates an ART system integrated directly with a taxicompany mobile or web application.

DETAILED DESCRIPTION

A video-enabled system that may be connected to a home, office or otherenvironment is presented, it is based on advanced computer visiontechniques generating data, which enables scene interpretation withoutthe need of video. Instead, visual serving may be used to create adigital representation of people or other things. The invention hasmultiple independent aspects and optional implementation features andeach will be described in turn.

FIG. 1 is a diagram with an overview of the overall system architecturefor integration in smart home and cloud solutions. The system is basedon Spirit, an embedded engine that analyses an input image andvirtualised the image into its digital representation enabling a digitalunderstanding of the scene. Spirit comprises dedicated IP blocks withembedded firmware. The system may comprise data receivers and devicemanagers (1), databases and storages (2) and smart-home interfaces (3).

The key features of this invention will be described in one of thefollowing sections:

1. Spirit

2. Alive

3. Art

4. Aware

5. JSON-Based Event Generation and Event Switching

6. Market Research and Application

7. Use Case Examples

1. Spirit

Apical has developed dual core technology models based on the humanvisual system. Assertive technology puts human vision into digitaldevices to create natural, seamless experiences everywhere while savingpower while Spirit technology mimics human visual processing, enablingdevices to understand what they see.

The performance advantage of the Spirit architecture over alternativesoftware-based approaches, whether measured in speed, power consumptionor cost is measured in order of magnitude. The algorithms on whichSpirit is based cannot be run in real time on any other existingprocessor architecture, including processors designed for computervision applications. Spirit also overturns the conventional process flowof sensor→image processor→video encoder→video decoder→video analysis.With Spirit, there is no need or function for any of the other parts ofthe system: they are expensive, consume power, actually reduce accuracy,and can be deleted. Spirit takes what was previously only possible on asupercomputer and puts it in a low-cost, low-power sensor.

1.1 Overview of Spirit

Spirit implements state-of-the-art computer vision and machine learningfully at the edge. It takes the most advanced techniques currently usedfor video analysis on server farms and embeds them inside a connectedsensor, operating with unprecedented performance and accuracy at verylow power.

Spirit can locate and track any number of people, their poses andgestures, from up close to far away, and in real time. Spirit takes asinput the raw data streamed from a standard image sensor.

Because of Spirit's unique architecture, there is no limitation on thenumber of people and their associated characteristics, which can besimultaneously monitored, nor is there any limitation on the distancefrom the sensor, provided the sensor is of sufficiently high resolution.

Spirit is an embedded engine, which converts a stream of pixel data intometadata describing the objects of interest within the scene, togetherwith their characteristics, as illustrated by FIG. 2. For example,Spirit coupled with a conventional CMOS image sensor can detect anynumber of people within a scene and continuously track their poses,trajectories and gestures. It does this by a process of virtualization,meaning extracting from the pixel stream all the basic objects for whichit has been trained to search, and distilling, meaning grouping theseobjects, determining characteristics such as pose, analysing theirtrajectories, and so on.

Spirit's output is a stream of metadata describing the instantaneouscharacteristics of each object in the scene, in a machine-readableformat ideally suited to subsequent data analytics (as shown in FIG. 2).This data is rich but highly compact, and represents a tiny bandwidthcompared to the source pixel stream.

Spirit can therefore be considered as a data compression engine capableof achieving a 100,000:1 compression ratio for the salient informationwithin the data stream while at the same time encoding it into a formwhich admits efficient subsequent analysis.

FIG. 3 shows a comparison between the compression from a Spirit engineas compared to standard HEVC for a raw pixel stream of 4K at 60 fps.While HEVC compression results in an enormous and unscalable cost ofanalysis, Spirit provides marginal cost of analysis due to its highcompression ratio.

Spirit operates in real time and can directly process raw image sensordata. Why process raw data, when virtually all conventional analyticsruns on video? First, because raw data contains the highest amount ofinformation and subsequent image processing may often only degrade it.Second, because this data can be characterized to deliver predictableaccuracy, whereas subsequent image processing may make suchcharacterization impossible. Third, it removes the need for the heavyon-chip and off-chip infrastructure for supporting creation, encodingand decoding of video. Fourth, because relying on video meansessentially that privacy is impossible.

1.2 Key Technical Characteristics

What is unique about Spirit is its ability to extract all salientinformation from streaming pixel data in real time without constraints.This is achieved by a combination of the following characteristics:

-   -   Real-time operation (for example up to 4K resolution at 60 fps).    -   Unlimited number of objects can be detected in each video frame.    -   Objects can be any size; they may vary for example from 50×50        pixels up to the resolution of the frame.    -   Complex objects, like people, can be composed by grouping        multiple component objects, like body parts.    -   Real distances may be estimated from the sensor location using a        calibration object of a known size.    -   State-of-the-art accuracy and predictability.    -   Many different kinds of objects can be searched simultaneously.    -   Operates on raw pixel data, or processed data.    -   Low power.    -   Low silicon cost.

1.3 Spirit for People Analysis

Spirit can be trained to search for a wide range of objects. For ART,these objects may be for example the component parts of a person:different positions of the head, upper body, full body, hands and so on.

Spirit employs a novel method for people analysis. People are detectedby a combination of up to 16 independent characteristics comprising thehead, head & shoulders and full body in different orientations, togetherwith additional models for hands. This approach yields multiplebenefits.

First, it dramatically increases the accuracy of people detection andtracking compared to methods that use single models for the entireperson. The method is robust to partial occlusions, where one part ofthe body is hidden behind another object, and is not dependent on theorientation of the person, who may have their back to the sensor. Themethod of grouping renders the system robust to errors.

Second, it enables extraction of rich information on pose. For example,an individual's head orientation, shoulder orientation and full bodyorientation can be independently evaluated.

Because Spirit can operate at high frame rates, people tracking becomereliable. Spirit continuously monitors the motion of individuals in thescene and predicts their next location. This analysis combined withother information extracted by the engine makes for reliable trackingeven when the subject is temporarily lost or passes behind anotherobject.

FIG. 4 illustrates an example of tracking and analysing a person inreal-time frame by frame for a sequence of two frames. A person isdetected and the characteristics of the head (head score and angle ofhead) and upper body (upper body score and angle of upper body) areextracted. The person ID is also shown with its associated confidencelevel.

FIG. 5 illustrates a further example where more that one person isdetected within a single frame. There is no limit to the number ofpeople, which can be simultaneously tracked and analysed. The number ofpixels in the frame dictates the smallest body part, which in practicecan be detected. Characteristics of the people detected may be extractedin real time, such as for example: person being tracked (1), head facingright (2), upper body facing right (3), full body facing right (4), headfacing forwards (5), upper body facing forward (6), full body facingforward (7).

The annotations from FIGS. 4 and 5 are generated using Spirit's metadatain order to give a visual reference.

1.4 Technology

Spirit may use a number of techniques to find objects of knowncharacteristics in each video frame. Most of the existing objectdetection techniques are based on machine learning systems, which focuson feature extraction and classification. Spirit may also use deeplearning techniques such as for example convolutional or recurrentneural network. Spirit may also use a hybrid machine learning techniqueor a random projection technique or another object detection algorithm.

Spirit may be trained off-line with thousands of examples of the objectsfor which it is to search. Spirit also has the capability to learnvariations of these objects, to a certain degree. Once it finds anobject in a given video frame, it looks for other related objects,groups them together, and then tracks them over time using predictivemethods.

Spirit is not designed to index all the objects in the world. It'sdesigned to detect, classify, track and predict common objects. Apicalconsiders the most important such object to be self-evidently theperson, but Apical has shown that animals, cars and other objects can beaccurately detected.

In addition to the core analytical algorithms, Spirit builds inproprietary modules, which control and characterise the image sensor. Asa result, the sensor is no longer employed as a conventional imagingdevice: it becomes a calibrated sensor with predictable influence on thequality of the virtualized data output by Spirit.

1.5 Performance

Spirit achieves unprecedented performance due to its algorithmic designand extensive implementation in dedicated hardware. This design achievesnear-100% utilization of computational resources, in contrast to the lowlevels of utilization typical of processors optimized for computervision tasks.

Spirit has an equivalent compute performance of over a teraflop, yetachieves this in a very compact and low power silicon core, capable ofimplementation in almost any device.

In addition, Apical has brought more than a decade's experience in imagesensor data processing, with the result that Spirit achieves a criticallevel of accuracy and predictability in a wide range of usage scenarios.

While Spirit employs dedicated hardware, it remains efficientlyprogrammable through its firmware layer. For example, different objecttypes are pre-trained and loaded as vectors into the engine in realtime, enabling objects as diverse as people, animals, cars and so on tobe detected, classified and tracked by the same core.

1.6 Implementation

Spirit comprises two components:

-   -   A hardware IP core performing the primary image analysis        functions, representing an area of dedicated processing on a        semiconductor chip.    -   An embedded firmware library executed on the same chip, running        on a processor of ARM M4 class or equivalent.

Because Spirit is uniquely able to process pixel streams without anyconventional video processing subsystem, it enables two classes ofdevice as shown in a smart camera, where video is virtualized anddistilled at the same time as it is captured and encoded; and a smartsensor, which contains no video subsystem but creates the same data fromthe same sensor. This is illustrated in FIG. 6.

In FIG. 7, the block diagrams of a standard video camera, of an ARTsmart sensor and of an ART smart camera are shown. The block diagram ofa standard video camera may comprise a standard ISP (Image SignalProcessing) block, which processes the output of an image sensor and anencoder. In comparison, an ART smart sensor may comprise a Spirit engineblock without the need of an ISP block and an encoder. An ART smartcamera may comprise a Spirit engine block as well as a post processingblock and an encoder.

1.7 Spirit in Context

Spirit uses the same kind of object recognition and deep learningalgorithms that the world's biggest technology companies are attemptingto deploy on supercomputers but implements them fully and withoutcompromise in a tiny piece of silicon inside a connected device. This ispossible only because of the highly optimized hardware-based design ofSpirit, which achieves 100% utilization of the chip resources. Theresult is performance, which is orders of magnitude higher than the bestof today's alternatives, and at orders of magnitude lower power.

Spirit changes the landscape for intelligent systems. Instead of theneed to ship video data from device to the cloud and process it withhuge computing resources, with all the associated problems of networktraffic, storage and compute costs, as well as privacy, an intelligentnetwork now needs only to transmit, store and process the virtualizedand distilled metadata which already provides a baseline description ofall objects of interest in the scene.

2. Alive

ALIVE is an intelligent eye embedded in Smartphones and other devices,which allows for video capture, search and publishing. ALIVEautomatically remembers the component of a video frame by frame whilebeing able to disregard the information or data that is unimportant.

2.1 Overview

ALIVE is a platform and a service for delivering new user experiencesaround video. An example of an ALIVE platform is shown in FIG. 8. Itconsists of one or more of the following:

-   -   Spirit engine in a SoC inside a smartphone or another device        such as for example a tablet or a computer.    -   Spirit engine in a FPGA inside a server, which may be linked to        a Commercial Content Distributer or to a smartphone or another        device.    -   ALIVE software App on a smartphone or another device such as for        example a tablet or a computer.    -   ALIVE software in a server.

ALIVE enables device-centric and service-centric business models:

-   -   for a device-centric model, the advantage is in delivering        better videos at lower bandwidth,    -   for a service-centric model, the advantage is enabling greater        usefulness of video, via searchability, and possibly a new kind        of way of sharing video.

Examples of key features are, alone or in combination:

-   -   Real-time indexing of video at capture time.    -   Capture best shots automatically.    -   Focus tracking.    -   Optimised encoding.

FIG. 9 illustrates the different elements of ALIVE, combining a state ofthe art hardware based object classification and tracking (A) to turnstill images and video into metadata, and a micro-cloud (B) consistingof a server based SDK performing behavioural analysis.

2.2 Technique of ‘Ghosting’

FIG. 10 shows an example of ‘Ghosting’, a unique technique that isemployed. The system does not need to capture or create photos or videosas it ‘Ghosts’ the video feed automatically into metadata, enabling amuch lighter weight cloud. Metadata retains only the positions andmovements of the people in the video clip including their gestures,postures and facing position and can also include further informationsuch as facial recognition and identifiers for example.

The fact that the technique relies on metadata has many privacyimplications. For example, the identity of individuals can be determinedbased on a digital fingerprint (for “face recognition”) as set by auser. No video or still images are required, as the Spirit sensor doesnot capture or create videos but instead just creates metadata from thescene.

As it is also based on Spirit technology, the ALIVE engine performsmassive data compression, as more content remains on the device; itstrips away the need for massive video file transactions and reduces thecost of video download, video search and online editing.

ALIVE may extract a wide range of metadata for scene interpretation.Examples are shown in FIGS. 11 and 12.

FIG. 11 shows a screen capture of a video where parameters are extractedin real time frame by frame at up to 4k 60 fps. In this scene, a widerange of information may be extracted, such as a person ID, the personposition and characteristics such as clothing information. Informationon the scene such as the presence of grass, foliage and sky may also beextracted in real time and may be displayed as annotations directly onthe image.

FIG. 12 shows another screen capture of a video in an unconstrainedenvironment where parameters are also extracted in real time frame byframe. Any number of people can be tracked accurately along with theirposes and trajectories. The engine may be able to extract people's bestthumbnails to be used for face recognition. Gestures are also detectedand tracked in real-time.

2.3 ALIVE Eco-Systems

Alive eco-systems may fall into the following categories:

-   -   Phone apps/personal apps: embedded video index search function        for personal real-time video content indexing, search and edit        functions.    -   Hardware acceleration: Apical Acorns: available as licensed        System-On-Chip solution for phones or as a data centre appliance        for in-line acceleration of video indexing.    -   Cloud Video apps: commercial Apps: servers operating with a new        type of video file distribution system that continuously        anticipates video index search requests.

ALIVE may provide the following functions:

-   -   Director: Automatically frames videos based on tracking specific        people; captures “best shot” still images.    -   Editor: Edits videos based on specific people, removing unwanted        parts and splicing together videos from different sources to        create a montage based on analysing peoples' poses.    -   Indexer: Adds metadata to video files describing who is present        and when.    -   My filters: when viewing shared videos, only videos or part of        videos that have people of interest are displayed.    -   Encoding: Spirit can produce reduced storage/bandwidth by        enabling ROI encoding at source or at transcoding stages.

Example use cases:

-   -   1. I shoot a video on a Spirit-enabled phone. During        capture, (i) my video is automatically framed in a chosen        “Director” style, (ii) a set of best stills are captured from        the video, (iii) metadata enabling indexing is attached. All is        uploaded to the ALIVE server.    -   2. My friend shoots a video on a non-Spirit phone. (i)-(iii) are        done by a Spirit FPGA on upload to the ALIVE server.    -   3. Some third-party content (e.g. football match) is streamed        through a Spirit FPGA to create indexing metadata.    -   4. Indexer allows me to search my video database for people I        care about, and go straight to where they appear.

Director may post-process my videos to improve their appearance. Editorlooks for similar videos from my friends (based on location and time)and locates people of interest within them. It then cuts several videostogether automatically by looking for particular poses of individuals(for example, looking at the camera) to create a single montage which isthen shared amongst the friends. When I access the video database ‘MyFilters’ only shows me videos with people I want to see. Friends canaccess the same content, but with their own filters.

A novel hybrid micro cloud app is built that sits amongst Smartphonesand the Internet. ALIVE eliminates the need to parse through a vastamount of videos to find someone or something in the past, thanks to areal time indexing of video while it is being captured.

The solution is available on consumer's devices with embedded softwarecombined with the micro cloud app. ALIVE also brings many social aspectswhere images blend with Internet of People.

The cloud app enables the following features, alone or in combination:

-   -   Video content is owned, captured, tagged, and indexed frame by        frame in real time.    -   Video may be indexed by people, names, and activities.    -   Video content is published in real time.    -   Digital history of unwanted scenes may be erased (discreetly and        in real time).    -   Control at atomic level with unique digital signatures within        frames.    -   Search for videos across one or more devices (for example on a        phone or a tablet).    -   Search video or clips with very specific people, posture and        things (for example I want to be able to find videos of my        family easily and jump straight to the best parts).    -   Search through a personalised filter, such as “my loved ones”        for example.    -   Feasible real video mashing.    -   Film, Gaming & Animation innovators introduce customer into        content applications frame by frame.    -   Mash up my living world with a virtual world where for example        “I become Mcllroy or Mickelson in Ryder Cup”.    -   Search for videos of “friends”.    -   Search for content archives.    -   Novel monetisation and advertising opportunities.

ALIVE reverses the costly cycle of Data Centre overbuilt to meetexponential demand for dumb video e.g. Facebook.

Along with search for personal content, it is also possible to searchfor online content. Content producers are able to capture and tag videosin real time production for their archives. ALIVE prepares for search byindexing video as they are recorded in real time and metadata is builtas video is being produced. As an example video may be analysed inreal-time from the source video stream and metadata may be createdthrough an Apical ACORN-1000 computer. Metadata may also be builtdirectly from video archives.

3. Art

The conventional concept of the smart home, on which the current productstrategies of all notable players are based, has a large hole in thecentre: the person.

The smart home debate centres on the competing standards formachine-to-machine communication, and on searching for purpose toconnect devices to the network. Most of the debate neglects the need forthe person to be part of the system.

There is a need to place the person at the centre of the system.Moreover, unless the system can respond to the person with very highaccuracy and predictability, the industry's technology push riskscustomer rejection.

Conventional technologies, from simple motion detectors to advancedvoice recognition systems, are totally unsuited to this problem. Theyare neither sufficiently accurate and predictable, nor do they provideanything even approaching the richness of data needed to interpretpeoples' behaviour and intent in order to provide practical intelligentdevice responses.

What is needed centrally in the smart home is a system for extractingrich information about peoples' behaviour and intent and delivering thisto connected devices in a consistent, contextualized and predictablemanner.

3.1 Overview

Truly smart homes must place the person at the heart and in control ofhis home. The smart home system of the future must be functionally rich,accurate and able to interpret person's behaviour and intent. Privacycontrol must be ingrained and no imagery should ever be formed. Withthese goals in mind, ART was designed to meet such needs. It is a newvision for future smart home.

-   -   ART is a revolutionary architecture for the smart home, based on        Spirit technology.    -   ART uses advanced sensors to create person's own digital avatar,        to which devices in smart home network respond.    -   ART unifies and enables diverse smart devices in providing        seamless, accurate and predictable response to everyone's needs.

ART is a novel architecture for the smart home. ART provides not just aplatform and protocol for describing the behaviour of people within thesmart home, it provides a unifying scheme enabling diverse devices toprovide a consistent, accurate and predictable experience to the userand a basis on which to represent the user to the digital world as theuser moves around the home and interacts with different intelligentdevices. ART may integrate home appliance sensors interoperating witheither single or multi-vendors solutions.

FIG. 13 shows a diagram with an example of an ART platform. ARTcomprises the following components:

-   -   ART Device firmware, which builds a digital representation of        each person within the environment, comprising the 4 key        descriptors: behaviour, pose, gesture and identity.    -   ART Hub firmware, which contextualizes individual ART local        representations to construct a global representation of each        person as they move through an environment of multiple sensors        in multiple locations.    -   ART Server software, which implements deep learning technology        enabling a particular ART installation to adapt to the        particular environment and user activities.    -   ApicaPs Spirit engine, a chip-level technology embedded inside a        sensor or smart home appliance, generating the primary real-time        data on which the people modelling is based.

These components together enable the entire smart home roadmap, fromindividual devices each of which is able to measure and respond to theuser's intent, to the fully integrated smart home supported by a networkof ART sensors continuously monitoring the patterns of your daily lifewithin it and driving the response of your environment to yourindividual needs.

ART is a fully saleable architecture solution for integration inSmartHome & Cloud solutions. ART supports local home solution and cloudservice delivery models with rich data analytics to enrich customerservice delivery experiences.

A Virtualized Digital Representation (VDR, =BNA) of a person comprises,as illustrated in FIG. 14:

-   -   Trajectory    -   Pose    -   Gesture    -   Identity.

A VDR is created on the hub by grouping and tracking the metadata outputfrom the Spirit engines. For privacy reasons, the identity of individualcan be determined based on a digital fingerprint (for “facerecognition”) set by the user. No video or still images are required, asthe Spirit sensor does not capture or create videos but instead justcreates

Metadata from the scene.

FIG. 15 illustrates an example of an ART home environment, whichcomprises one or more ART spirit-enabled sensors. ART creates a digitalrepresentation of each person using one or more ART sensors. ART thenuses this digital representation to analyse trajectory, pose, gestureand identity and provides a platform upon which smart home devices canrespond to and predict the person's intent and needs.

FIG. 16 illustrates that using Spirit, it is also possible to enable anew system avatar for the user-centric smart home. ART can effectivelycreate a visual DNA of the home individuals, and create a working avatarin the home to which devices and smart home network can respond to.

3.2 How Does ART Work?

Using the virtualized data Spirit extracts from the raw sensor feed, ARTbuilds a virtualized digital representation of each individual in thehome. This representation comprises each individual's:

-   -   Trajectory around the home, including for example the actions of        standing and sitting.    -   Pose, for example in which direction the person is facing, and        (separately) in which direction they are looking.    -   Gesture, for example motions made by the person's hands.    -   Identity, namely the ability to differentiate between people and        assign a unique identity (name) to each person.

Why these 4 descriptors? Because any relevant behaviour may be describedin terms of these 4 descriptors. For example,

-   -   Trajectory+Pose=Intent

Smart home understanding: My smart device should wake up when I approachand look at it.

-   -   Gesture+Pose=Focus

Smart home understanding: A smart light switch should respond to a handgesture if I am also looking at it.

-   -   Identity+Gesture=Control    -   Smart home understanding: I want to turn on/off smoke alarm, but        I don't want my kids to do so.    -   Trajectory=Flow/Occupancy    -   Smart home understanding: The home can learn at what time I come        downstairs on which day of the week.    -   Trajectory+identity+(Pose)=Behaviour    -   Smart home understanding: Do my kids spend too much time sitting        in front of (watching) the TV?    -   Abnormal Trajectory+time=Care    -   Smart home understanding: Uncharacteristic behaviours.

FIG. 17 illustrates ART's device API from which this representation ismade available continuously to any connected device, such as a localappliance. These real-time APIs deliver a standardized description ofthe person, known as the ART Protocol, upon which different devices canreact in a unified and consistent manner.

Predictable accuracy has been stressed repeatedly: without it, a stableand responsive smart home will never be possible and the problemsexperienced with introduction of technologies, such as voice recognitionand gesture detection, in related markets will be magnified to theextent they become showstoppers. ART's construction builds inpredictable accuracy at every stage, from the raw data analysis upwards,enabling Apical to validate the performance of a specific applianceindividually and as a component of the connected system.

FIG. 18 shows a diagram representing the systematic approach to accuracyand predictability that is deployed in order to process people behaviourin real time. The ART sensors or cameras incorporate both the Spiritsilicon core and the ART embedded firmware, and in essence turns the rawdata from the image sensor into individual's descriptor such astrajectory, pose, gesture and identity. At the higher level, the ARTcontrol software is embedded into the ART controller, whereinindividual's descriptors are being managed further. In summary, all ofthe sensor metadata are fed onto an ART controller, wherein the ARTcontroller becomes the home's “ART Heart”. The ART firmware uses aproprietary protocol to create ART events. ART events are then createdwith the local network, and the ART firmware is able to locate a numberof device options.

ART may use conventional low-cost image sensors as found in smartphonesand home monitoring cameras. ART devices may also guarantee that noimages or video are extracted from the system—because they are nevergenerated.

ART events are created locally. One of the advantages of the ART controlfunction is that it can be invisible within existing home hubs, such assecurity panels, sensors, smart TV's, Wifi routers, HEM systems, gamingplatforms or light bulbs etc.

The ART Heart controller pushes ART events to the home's smart devicesnetwork as specific commands. The ART event streams can further be sentto cloud analytics apps, such as cloud-based data monitoring, datagathering or learning service. This is illustrated in FIG. 19.

Raw sensor data may be processed and continually streamed to the ApicalHub Software. The ART Hub software processes the data locally and pushesout a heartbeat to the cloud. The Heartbeat may contain severalservices, such as “presence”, “recognition”, “movement”, “gesture” and“mood”. The HeartBeat can be sent in real time to a specific device ordevice controller. The ART heartbeat may be pushed to the local smartdevice network as specific commands. ART is then able to control orvalidate the performance of specific devices or appliances on the homenetwork. This enables a person centric control of smart devices.

3.3 Why ART?

ART enables the injection of the following enhanced intelligence:

-   -   ART always knows who you are.    -   ART powers up when you arrive and down as you leave.    -   ART intuitively learns adjust to all the things you like.    -   ART solves all the false positive alerts you send me.

A smart device must understand if a person is paying attention to it, ifthat person is authorized to interact with it, if a gesture correspondsto that person, and whether it is responding in a way, which pleases orirritates that person. The set of connected devices comprising the smarthome must be able to monitor, contextualize and predict the behaviour ofpeople as they move throughout the home. And they must do so with aguarantee of privacy, meaning no imagery is generated and the user is incontrol of the data that is collected.

The ART sensor may also be programmed to recognise a specific person'sgesture but not someone else's. ART continuously analyses each person inthe room and interprets their behaviours as events. ART understands awide range of behaviours, from counting the number of people in the roomand determining where they are moving from/to to specific gestures byidentified individuals. The sensor can also track the way the person isresponding to the sensor, such as if the person is comfortable or notcomfortable for example.

However, ART is not just about facilitating the connected home. Itoffers a new vision of the smart home. This is not about clever deviceslooking for problems to solve. This is about providing you, the personat the centre of it all, with a faithful companion who can learn aboutyou, respond to your needs, and enrich your daily living experience.

Spirit's use of raw data directly from an image sensor sets it apartfrom conventional techniques, which rely on post-processing of video.This has a number of benefits. First, use of the raw data enables thehighest degree of accuracy and predictability. Second, system cost andpower is dramatically reduced, as there is no need to generate, compressand process actual video. Third, because no video is created, there isno possibility for a third party to extract images from a person'sprivate environment.

3.4 Privacy

Just as vision is our primary sense, video contains the richest datacurrently available for analysing peoples' behaviour. However, the levelor granularity of data contained in video is obviously not adjustable:it carries too much information.

For example, you may be comfortable installing a video camera above yourfront door, but you are probably not so installing one in your bedroomor bathroom. General privacy concerns about who has access to your mostintimate data apply strongly here, and recent events such as the hackingof baby cameras highlight the problem. Users are aware that softwarecontrols and encryption are inherently vulnerable, especially if theyrequire manual setup and maintenance: many people have no security ontheir home wifi routers not because they are unaware of security, butbecause setup is not always reliable. ART does one important thing inthis space: it enables the creation of smart sensors using visualinformation, but crucially which never form imagery or video at ahardware level. It is demonstrably impossible for any third party toaccess imagery from such a device. Of course, there remains an importantdebate about how the rich behavioural information extracted by ARTimpacts on privacy. This is a topic of comparable complexity to webprivacy.

However, because ART provides a highly structured, hierarchicaldescription of behaviour it provides an excellent platform on which tobuild systems where the user is fully in control of their data. First,because all data can be managed locally within the home network. Second,because the granularity of the data can be easily selected appropriatelydepending on the application, from the most simple “there is a human inthe home” to the more rich “Michael is in the living room andinteracting with the lighting system”.

Since ART also uses the technology that is used in Spirit, it can alsoemploy the technique of ‘Ghosting’ (as shown in FIG. 10). Hence ARTenables the homeowner to be in complete control of functionality andprivacy. No imagery is created with Spirit at a hardware level. Spirit,if enabled by the homeowner, takes a selection of facial featuremeasurements. ART ID is then undertaken by comparing data to a set offacial measurements held locally.

Despite the fact that ART is able to recognize individuals, ART is alsoable to guarantee privacy because at least of the following:

-   -   ART ecosystems do not create any imagery of any kind; no video        is used within the ART system.    -   ART generates rich data on individuals, and if enabled by the        user, their identities. However, the critical point is that,        unlike a video camera, the user is able to control how much data        is captured. For example, whether individual identities are        captured, granularity of detection (“there is someone in the        room”, “someone is walking towards the kitchen”, “Michael is        walking towards the kitchen”)        -   A good analogy is the iPhone's fingerprint sensor. You don't            need to use Touch ID to unlock your iPhone, and, if you do,            it doesn't store/transmit your fingerprint into the cloud.        -   In effect ART offers the homeowner a control solution that            has “self encrypted privacy”.

3.5 ART Sensor Locations

There are a large number of possible locations inside a home or officeenvironment to place the smart ART sensors. It is important that thesensors are placed in the most convenient possible place, such that thesensors are easy to install and that they provide the best possibleaccuracy.

Examples of locations where sensors can be incorporated are for exampleon or in relation to a light bulb or a light switch.

Light switch may present a very convenient location where a Spiritsensor can be incorporated due mainly to the following reasons:

-   -   power is already available,    -   location is ideal for performance and accuracy.

FIG. 20 shows a diagram of a light switch incorporating the followingelements:

-   -   a lens and sensor assembly (1),    -   a chip (2) capable of detecting objects from the sensor data,        wherein the chip outputs a data stream describing people, their        locations, trajectories, identities and poses,    -   a means of delivering the data (3), such as a wireless        transmitter, which may be integrated into the chip,    -   one or more IR LEDs (4),    -   a lighting control (5),    -   and an indicator LED (6) showing a different colour when the        detection chip is operational and not operational (privacy        indicator).

In addition, the light switch could also incorporate a physical shutterin order to cover either a portion or the entire ART sensor. This couldbe implemented for privacy reasons as an additional security that thevideo is not being recorded or analysed.

In particular, a home automation system incorporates the following:

-   -   one or more light switches as above connected to a local        network,    -   a software application resident on another device connected to        the local network, capable of parsing the data from one or more        sensors and generating information about the people in the        environment,    -   a software application resident on the cloud accomplishing same,    -   where the information consists of a time series of events based        on the data, such as “a person entered the room”.

The sensor may be an image sensor, such as a conventional Bayer CMOSimage sensor. The lens may preferably be wide angle (up to 180 degrees).The lighting control may be for example a physical switch, rheostat,capacitive sensor etc. The IR LEDs provide scene illumination indarkness.

3.6 Configuration

The ART platform places the homeowner at the centre and in control ofhis smart home. An ART home setup may consist of a network of sensorsthroughout the home with a combination of simple sensors and ID sensors[people recognition sensors]. An ART hub would also be incorporated in a3^(rd) party Hub that is able to communicate with the Cloud (asappropriate) and other smart devices.

Set up procedure: setting up the ART system may consist of three stepsusing the ART app—all undertaken on a computer, smart phone or smarttablet. The ART app is setup to guide the homeowner through a simplesetup procedure, as follows:

a) Configuring the homeowner home

The homeowner will need to download the ART app onto for example asmartphone or tablet, they will be prompted to follow the procedure:

-   -   1) pair the sensor with the smartphone,    -   2) the ART app will direct the owner to walk to each corner of        the room in question, and relevant doors, windows and other        areas of interest ,    -   3) once one room has been configured, the ART app will prompt        the homeowner to move onto the next room to be configure,    -   4) adjacent configured rooms are linked (again through prompting        from the ART app),    -   5) install the smart door bell and follow the ART app prompts on        the entry and exit points.

The homeowner will then be prompted to provide his or her name andfamily names as well as house chums names. Next, the ‘who lives here’button would need to be selected, with the option to take a full frontimage and to enter a name for the image. Select ‘who lives here’ buttonon the ART app, and take a full front picture and then name the image.

b) Connect and configure SMART devices with ART

The ART system is compatible with many of the smart home devices on themarket today and ART can be used to control and manage the completesmart home, place the homeowner at the heart of the home and fully incontrol. Within the ART app, select ‘configure compatible devices’—theART app will list the compatible devices.

The following steps may configure a device within the ART app:

-   -   1) select the device from the list,    -   2) select the nature of the ART control wanted for the device,    -   3) the ART app will confirm configuration has been successful.

Examples of devices that can be configured to the ART app include, butare not limited to:

-   -   Smart lighting systems (individual lights).    -   HEMS.    -   Alerts.    -   Kitchen appliances.    -   Danger zones.    -   Garage doors.    -   Smart door bells/entrance doors.    -   Baby/elderly alarms.    -   Security cameras.    -   Pet monitoring products.    -   AV/media.

c) Switch your ART system from configure to live

Once devices have been configured, the last step is to select ‘Go Live’on the ART app. The ART system will now be live.

3.7 Sensor Fusion

In order to provide an even richer set of information, it is alsopossible to use information provided by multiple sensors.

An ART sensor may be configured alone for room occupancy and lightefficiency data for example. It may also be configured in conjunctionwith a temperature sensor in order to perform various functions such asroom/home thermal rating, heating efficiency or manual guidance on roomheating control. Additionally, it may also be integrated with smart TRVor with power room usage.

As another example, a wearable device could also be used and linked withvideo.

A trivial example is given:

-   -   person enters room, wearing smart watch;    -   face sensor, make gesture (e.g. wave);    -   Spirit correlates motion detected by smart watch with image        data, and “pairs” person (i.e. the ID we give them).

The method developed is equivalent of “pairing” (as per WiFi andBluetooth), but in the context of a scene coming from Spirit. This couldbe an important part of all systems trying to fuse visual data (andsensors based on Spirit) with other types.

Re-pairing may be done when a connection is dropped. Other classes ofdevices may also be used, for example, a car driving past could be ID'dbetween the image data and its own motion data. ART may also be fusedwith voice control in order to enhance user experience of voice actuatedcontrol systems.

An example of enhancing a user experience with the existing Amazon Echodevice is given:

-   -   Glowing light indicates Echo is ‘awake’:        -   Only listens with person intent,        -   Avoids the need to say “OK, Alexa”.    -   Face recognition:        -   Alexa able to address individuals,        -   Automatic logging into person's amazon account,        -   Tailored up-sell offers,        -   Personalization—Echo can play your playlists.    -   Control:        -   Echo ignoring childrens' requests.    -   Gesture:        -   Control Echo volume or turn Echo on/off.

In essence, ART provides “focus” in the sense of mouse focus on adesktop, while voice recognition like Siri provides the equivalent textinput. This solves the problem of differentiating voices and figuringout if a spoken word is directed at the device.

3.8 ART CCTV Security

Current available IP cameras are not compatible with a scalable domesticCCTV proposition as illustrated in FIG. 21. Current CCTV optionstherefore have to sacrifice with resolution and frame rates that are atsuboptimal levels as illustrated in FIG. 22. This is in part due to theupload bandwidth restriction of standard CCTV systems.

ART proposes a unique solution to enable smart security CCTV cameras tohave significant benefit. ART offers the following advantages enablingCCTV security to enter the Cloud product:

-   -   ART reduces bandwidth load requirements. Hence Cloud CCTV        storage becomes a reality.    -   ART enables automatic crop and zoom.    -   ART enables ID and thumbnail capture.    -   ART enables car model and #plate capture.

An ART enabled CCTV system enables significant consumer functionality aslisted in Table 1 leading to rapid product adoption.

TABLE 1 Functionality comparison between traditional and ART CCTV systemTraditional Domestic ART enabled CCTV, DVR CCTV System with cloudfunctionality Camera Resolution Disk size restricted High definitionBandwidth Requirement Too restrictive Ultra low requirement StorageLocal Cloud Indexation Motion Event Alert Functionality Simple ComplexCompatibility Limited High ID No ID ID, #plate, unknown

FIG. 23 illustrates an example of the different steps in which an ARTsecurity system is enabled. First, ART CCTV solution dynamicallypreserves resolution of selective areas (or region of interest) withineach frame. ART then dynamically adjusts the effective frame rate basedon content and manages streaming level based on user control. The outputstream is then fed into ART dynamic control for further optimization andcan be analysed in the cloud.

The upload bandwidth restriction previously illustrated in FIG. 22 cantherefore be removed entirely.

Along with ART CCTV solution, ART can also provide services for a widerange of products as listed in Table 2.

TABLE 2 List of services and products that ART can provide. ProductOffer ProductDetails ART intelligence Revenue Options Smart ART embeddedrange Presence detection Base package plus ART of domestic smartreducing bandwidth Tiered pricing structure: CCTV external CCTV cameras.requirement Cloud based storage of Indexation of X cameras triggeredfootage video capture; ID Period/volume storage opts Cloud basedanalytics and #plate ID/#plate capture of stored images Advance alertPet notification Intelligence undertaken functionality Advanced alertsby ART Heart Control Smart ART enabled smart Active presence Basepackage plus ART door bell system detection Tiered pricing: Entrance ARTenabled garage ID/#plate ID/#plate door control capture / indexationAlerts ART Heart Control Alert functionality Unknown callersnotification Connectivity with intelligent CCTV Smart 6 smart ARToccupancy Advanced HEMS Share of potential property ART sensors (nocamera Entry platform for energy saving Energy functionality) advancedSmart Tiered pricing: ART Heart Control Home control Device control; OTT(e.g. music) Smart Smart ART sensors Monitor the health Care package,covering Care ART Heart Control and wellbeing the Activitylevel/movement Platform Optional Smart internal home occupantsmonitoring Camera Trip alerts

Within the home, applications can range from Security around the home,HEMS, to Care and Control. And ART security CCTV can further bring theadvantage to eliminate false positives, to provide workable streamingspeeds, and to advance functionality and finally to transform consumerexperience. An ART enabled CCTV system enables significant consumerfunctionality, leading to rapid adoption.

A market leading proposition to place the consumer at the heart of theirconnected environment is now possible:

Intelligent Security:

-   -   Real time data,    -   Identification and notification,    -   Rich functionality (ID and #plate capture).

Existing service enhancement:

-   -   Smart HEMS,    -   Care Family alert service,    -   Video functionality—dynamic encode,    -   3^(rd) partnership models.

Third party OTT service for companies to benefit (with service providercustomer's permission), the event data stream from the ART sensornetworks, for example:

-   -   Smart Thermostats (e.g Nest),    -   Music streaming (e.g. Sonos),    -   Google,    -   HUE,    -   Xbox.

3.9 ART Smart Application

The number one customer issue with IP security cameras remains falsealarms. As an example, false alarm messages from a single IP security UIcamera proposition from a major US service provider resulted in thedetection of more than 120 false alarms in a 3-hour period, equating tomore than 1,000 false alarms in a 24 hour period.

ART smart application is developed to minimize the number of falsealarms. ART smart sensors transform the consumer experience by placingthe ‘person’ at the Heart of the home, and the smartphone app is simplya communication tool.

FIG. 24 shows an example of a screen shot of an ART smart application,in which a dashboard let's individuals see what is happening inside ahome environment when they are away. Multiple functions may be availablesuch as, for example, but not limited to:

-   -   Monitor: see what's happening at home now and what's happened        recently by looking at different category groups.    -   Control: easily control your lights, locks, electronics,        appliances and other connected devices in your home from        anywhere.    -   Customize: set your connected devices to work in new ways when        your needs change.

Features available through the ART application include:

1. Identity

-   -   Phone App (PA) captures face using front camera as target for        face recognition (FR),    -   PA tells user to look at camera and press button to capture        face,    -   During setup, user looks at camera and makes a gesture to        trigger face capture.

2. Field of view

-   -   PA indicates to user their approximate location as they walk        around a room within the field of view (FOY) of the camera.

3. Orientation

-   -   Establish orientation of the camera through (a) accelerometer        or (b) relative orientation of a detected person, typically        within 90 degree rotation.

4. Multi-camera calibration

-   -   Establish spatial relationship between different views of the        same person,    -   Person may stand within FOV of two cameras, press a button on        PA, instructing system to locate the same person within both        FOVs.

5. Approx. 3D reconstruction

-   -   Could edge detection be used to detect wall edges/corners and        establish approximate cuboid of room?    -   Then coupled with object size, perform an approx. 3D        reconstruction?

3.10 Natural interactions with the sensors

Another important aspect of the invention is the ability to know whenthings are not understood. This may be implemented through a natural wayof interacting with the sensors.

For example, there is no need to walk towards the ART sensor to activateit, instead the activation of the ART sensor may be done automaticallywhen an authorised person enters a room and quickly glances at thesensor in order to activate and access it. This may be implemented bytracking automatically and in real-time as soon as the person enters aroom. From the real-time tracking, the position of the face iscalculated or estimated. Real distances, which may then be used to knowthe position of the face and location of the person, may be estimatedfrom the camera position. As the face appears within range of the sensorthat needs to be activated, the face recognition engine is activated.This can be done using one or multiple sensors.

Sensors may also be de-activated from a seamless interaction with anauthorised person. The API provides a descriptor of each individual inthe environment. On top of that, it provides prediction abilities, notjust reformatting frame-by-frame data.

The interactions with the sensors and the interface of the sensors aresimple enough to be used by a homeowner, with no necessary calibrationneeded.

3.11 Software Architecture

FIG. 25 shows a diagram of ART 3-Layer Software architecture. Thesoftware architecture that underpins the ART system models closely thedeployment of multiple sensors in a Smart Home in 3 layers:

-   -   The Edge Layer processes raw sensor data at the sensor itself        and is the core engine of the system. The Edge Layer creates and        manages all services, handles external interfaces and includes        cloud applications.    -   The Aggregation Layer provides high level analytics by        aggregating and processing data in temporal and spatial        (including multiple sensors) domains. The Aggregation

Layer collects multi-sensor data, delivers class of service switching,and manages Edges.

-   -   The Service Layer handles all connectivity to both the Smart        Home controllers and to the end customers for configuration of        their home systems and the collection and analysis of the data        produced. The Service Layer collects single-sensor data,        produces sensor meta-data and controls device mode of operation.

The Edge Layer

The Edge Layer consists of a layer of embedded software that sits at theedge of the Smart Home network of sensors—that is right at the sensoritself. It also contains Spirit. As stated previously, Spirit isat-the-edge technology, which virtualizes video or raw sensor data froma scene directly into a digital representation of all its importantfeatures. This virtualized data is then distilled into a digitalunderstanding of the scene.

A Spirit-based system, for instance, can monitor the behaviour of allpeople within the scene based on pose, movement and identity in realtime, at up to 4K resolution. Spirit comprises dedicated silicon IPblocks with embedded firmware.

The remaining parts of the Edge Layer perform the necessary managementand control functions that allow the system provider to monitor thesystem for faults or change the behaviour of the data processing in thesensor. All of this happens in real-time and the output is a set ofsensor Meta-Data that is pushed up into the Aggregation Layer.

The Aggregation Layer

The Aggregation Layer takes the metadata produced by the Edge Layer andanalyses it further, often combining multiple sources of data togetherto create events as functions of time.

This layer is even more sophisticated, because it can also interpret aset of rules for the creation of events—rules that have been injectedinto the system through a suite of applications for the user of thesystem.

Finally this layer prepares the events for delivery as a service, whichincludes scheduling algorithms that drive a multi-class of service eventswitch before passing the event data through to the Service Layer.

The Service Layer

The Service Layer allows the system to interact with the outside world.The outside world in this case could be an Energy Controller within thehome or an Entertainment system. It could also be a Home Security systemor a Fire and Safety system. These real-time control systems subscribefor an event service that is packaged, delivered and monitored by theService Layer.

The end user or homeowner can also use the applications within theService Layer to input data and parameters about their home setup. Theycan use the applications to provide the system with a personal model oftheir environment for example, or to specify behaviours and rules forthe combination of the sensor data with other systems in the home. Thereare also applications that aid the homeowner in setting up the systemand walking them through the various steps involved, prompting them toenter data as required.

Finally, the Service Layer also contains data analytic tools that can“pass-through” data from the home and store it in the cloud. From there,it can be analysed through a set of web services Application ProgrammingInterfaces (APIs) remotely.

3.12 Flexible Architecture Multiple Deployment Options

It is important that the ART architecture is flexible enough to bedeployed in a variety of physically distributed platforms. For example,not every home will have the same devices and gateways that connect itto the internet. Certainly every home will have a differentconfiguration of sensors and controllable systems.

Therefore the architecture was designed so that the modularity andinternal interface designs allow a variety of distributions to happen.In particular, there are 3 main options for the distribution of thesoftware across platforms as described individually below.

Option 1: Hub Device & Cloud

The first option is illustrated in FIG. 26, and it shows how a largecomponent of the overall system is self-contained in a single Apical HubDevice. This is particularly useful in the situation where sensors orcameras are already installed and the user simply wants to make thosecameras work in a smarter way, and feed into a next generation of SmartHome controllers.

With this option, all 3 layers of the architecture are contained withinthe Apical Hub Device, with a portion of the Service Layer stillremaining in the cloud. Note how the internal interfaces in the ServiceLayer readily prepare the software for deployment in this scenario asthe interface is defined as a Web Services interface, effectivelyinterlinking internal software modules in the mode of a “micro-services”architectural approach.

Option 2: Sensor, Home Gateway & Cloud

A second option is illustrated in FIG. 27. Here, the Apical Spiritcomponent is now physically embedded into the sensor/camera deviceitself within an ASIC (Application Specific Integrated Circuit). Asilicon device contains the edge layer functionality and processes thesensor data into metadata in real-time at the sensor itself.

The second component of the Edge Layer then sits in a home gatewaydevice. This device could belong to a third party such as anentertainment company or an energy company. This component would bedeployed through a partnership and co-development whereby the ARTsoftware resides on a virtual machine running a small server within theexisting software system of the device.

The gateway or hub component of the Edge Layer is used to centralisesome management components of the architecture rather than replicatethem across all of the sensors themselves. It aggregates fault andmanagement data and does some analysis before pushing it up to theAggregation Layer.

The Aggregation Layer sits fully in the Home Gateway Device, takingmetadata from the Edge Layer and analysing and synthesising events thatcould be used as part of the various services offered out. There is alsoa part of the Service Layer residing in that device, so that events canbe sent directly to Smart Home controllers within the home rather thanup through the internet connection.

The final Service Layer piece is similar to that described in Option 1above, and resides in the cloud. Note that raw sensor data and metadatacan be optionally “passed through” the Aggregation Layer and into thecloud Service Layer for storage and non-real time analysis and usage.This functionality provides the means for end users and partners tovalidate and audit service data, or simply record the data in a securelocation.

Option 3: Sensor & Cloud Only

There may be situations where there is no home gateway or hub devicelocally available from a partner to host the Apical software. In thisinstance, the entire 3-layer architecture can reside in the cloud asshown in Option 3 as illustrated in FIG. 28. Note that there is stillsome embedded silicon and firmware in the sensor which acts as the mainportion of the Edge Layer, but the Edge Layer management and controlfunctions are now in the cloud.

In this scenario, the operation of a 3rd party controller will bedependent on the internet connection for it to operate successfully andreceive any pushed service events from the Service Layer.

With the 3 options provided, most deployment scenarios are covered andthe software architecture has an in-built mechanism to scale and tocover a wide variety of market use cases. This architectural approachalso enables multiple partners to avail of a core piece of sensor IPR,which is designed, developed and supported centrally by Apical. Forexample, Energy, Security, Safety and Entertainment providers can allavail of similar service events and thereby share the cost of supportingthat technology base over the expected lifetime of the Smart Homeindustry, which should extend for 10+years in its first generation ofproducts.

3.13 ART Detailed Software Architecture

Orthogonal 3-Plane Design

The internal structure of the architecture is primarily designed to bemodular and in particular, to separate out orthogonally the data plane,control plane and management plane. These planes cut through the 3-Layerarchitecture as shown in FIG. 29. Effectively they represent a groupingof similar types of functions in each layer so that independentdevelopment roadmaps for each plane can be supported. However theprimary reason for this aspect of the architecture is robustness.

The data plane is the set of functions that receive raw sensor data,process it into metadata, and produce service event data to push out tovarious controllers. The control plane represents a means toparameterise the performance of the system in real-time, for exampleadapting the type of metadata being requested, or the performance of thesystem.

If a 3rd party control system requires a high level of robustness fromthe event production, then should a problem arise in the control plane,it should not affect or degrade the performance of the data plane. Thisis what is meant by orthogonality—an attempt for complete independenceof operation to ensure that there are no hard crashes of the systemshould one component degrade for whatever reason.

The management plane is recording and analysing performance monitoringpoints within the system, fault detection and error masking andreporting. It is also responsible for recording and persisting themanagement data for support, auditing and billing functions. It isdesigned to operate independently to the other 2 planes so that if aproblem arises in the management function, the controllers that aredependent on the data will still receive that data without a hard systemcrash.

Reliability and robustness across multiple hardware platforms is achallenging task, particularly when multiple vendors are involved. Thesearchitectural choices have been made to specifically reduce the risk ofhard crashes in feeding important security, fire, safety andhealth-critical functions in the Smart Home.

Modular System with Well-Defined Interfaces

Each of the 3 Layers of the system is divided into sub-layers. Thisbreaks down the entire architecture in the components shown in FIG. 30in which external interfaces and internal interfaces across the threeplanes of operation are also shown. The components are created by acombination of the various layers, sub-layers and planes in the system.

The Edge Layer has 2 Sub-layers:

-   -   The Edge Management Sub-Layer which houses the control and        management functions of that layer.    -   The Edge Device Sub-Layer, which resides on a sensor device.

The Aggregation Layer has 3 Sub-Layers:

-   -   The Edge Receiver Sub-Layer that communicates with the Edge        Layer and manages that interface and its performance, while also        performing some analysis of metadata received.    -   The Analysis & Synthesis Sub-Layer combines and analyses        metadata and environmental information on one or more sensors        over time to synthesise events which are then passed upwards        through the system.    -   The Service Preparation Sub-Layer combines event data received        from the Analysis & Synthesis Sub-Layer with metadata, Service        Rules and customer-centric environmental data to produce, switch        and deliver service events.

The Service Layer has 2 Sub-Layers:

-   -   The Hub Sub-Layer is the part of the Service Layer that is        located in the home itself, residing on a home gateway or hub        device either supplied by Apical or by a third party, or a        device that is already performing an existing function such as        router, set-top box or similar device.    -   □The Cloud Sub-Layer is the part of the Service Layer that is        located in a datacentre cloud environment connected to the home        through the Internet.

Finally, there are a series of well-defined interfaces interconnectingthe various modules so that each module can be implemented using methodshidden from the other modules. This ensures that a complex system canhave roadmaps for the development of individual components with aslittle coupling as possible in order to ensure fast reaction time tomarket conditions and new ideas in the Smart Home industry—particularlyimportant given the fact that the Smart Home is an emerging market.

Many of these interfaces are designed as web services interfaces so thatthe components do not necessarily have to reside on the same physicalplatform, so long as there is a standard IP network available to connectthem e.g. a home wireless network or the internet.

Detailed Architectural Components

Each layer and plane describing the architecture may comprise severalselected functions. Some key examples of functions are presented in FIG.31.

Examples of functions present in the service layer within the data planeare: database management system (D1), query, search and retrieve API(D2), off-line analysis application and big data tools (D3), eventservice delivery external controllers (D4), data upload (D5).

Examples of functions present in the aggregation layer within the dataplane are: event generation for services (D6), event queuing andswitching (D7), source fusion (D8), scene analysis and synthesis (D9),API for third party plugins (i.e. face recognition) (D20), meta-datapre-processing (D11).

Examples of functions present in the edge layer within the data planeare: meta-data extraction and analysis (D12), external sensor interface(D13), data upload (D14), metadata extraction and analysis (D15), dataupload (D16).

Examples of functions present in the service layer within the controlplane are: control and configuration API and apps (C1), control andconfiguration API (C2), service delivery and data upload configuration(C3).

Examples of functions present in the aggregation layer within thecontrol plane are: scheduler control (C4), control and configuration API(C5). Data processing parameters and configuration (C6), environmentalmodel control parameters (C7), edge interface configuration (C8), edgelayer control (C9).

Examples of functions present in the edge layer within the control planeare: control and configuration API (C10), data processing parameters andconfiguration (C11), data processing parameters and configuration (C12).

Examples of functions present in the service layer within the managementplane are: management API and applications (M1), management aggregationand edge layer (M2), service interface monitoring (M3).

Examples of functions present in the aggregation layer within themanagement plane are: service preparation monitoring (M4), managementAPI (M5), analysis and synthesis monitoring (M6), edge layer management(M7), edge interface monitoring (M8).

Examples of functions present in the edge layer within the managementplane are: management API (M9), update manager (M10), self-test andperformance monitoring (M11).

Note that this diagram does not represent a complete set of thefunctions involved, further examples of functions are for example, butnot limited to: data persistence, database storage and lower leveldrivers.

Where functions could potentially straddle two or more components of thearchitecture, a deliberate choice is made to refactor that module sothat it no longer straddles multiple components. In this way,architectural integrity can be maintained as the priority over thereplication of some data or lower level functions or libraries.

FIG. 31 is presented here to illustrate the next layer down of thearchitecture definition. It should be clear from this diagram thatbuilding a robust, reliable and multi-vendor event service for the SmartHome is considerably complex and made even more so because of themultiple possible options for physical deployment and underlyingplatform variability. However with strong architectural features andchoices, the ART software architecture is built for the long-term futureof the Smart Home industry.

3.14 ART Deployment

FIG. 32 shows a diagram with an example of an ART deployment via aSpirit-enable SoC, a camera CPU and a hub.

4. Aware

4.1 Overview

AWARE is a platform for converting peoples' behaviour into big data. Itconsists of:

-   -   Spirit engine in a SoC inside a camera or sensor,    -   And/or Spirit engine in an FPGA in a router or server,    -   AWARE server containing database(s), business logic, and        interfaces to client applications.

FIG. 33 shows a diagram of a Spirit embedded architecture as implementedin an AWARE platform. AWARE image processing hardware block comprises aconfiguration block, controller, filtering block and classifier block.Both the filtering and classifier block subscribe to an AXI busconnected to the DRAM and a CPU, which contains the AWARE firmware. Thefirmware extracts and transmits metadata to an API, network, file or adatabase.

FIG. 34 shows a diagram of the architecture of an AWARE cloud SDKplatform. ART devices may send video or metadata to a metadata databaseor video database located on a server. They may also send directlymetadata to the AWARE SDK platform where analytics are performed (forexample scene, event or multi-camera analysis). Search retrieval mayalso be performed as well as other high-level complex or 3^(rd) partyprocessing. The AWARE cloud SDK platform may also communicate with acustomer application or web interface.

An essential aspect of the invention is the creation of a Track Record,which is the reformatting of real-time metadata into a per-object(per-person) record of their trajectory; pose (and, possibly, identity).The Track Records are stored in a MySQL-type database, optionallycorrelated with a video database.

The AWARE server performs the following pseudo real-time functions:

-   -   Population of a database of Track Records,    -   Post-processing of Track Records to correct for lost/swapped        objects and multi-camera tracking,    -   Provision of higher-level tracking information to third party        applications,    -   Adaptive learning.

Privacy: Video database is optional. Some applications, like securityapplications, require it; others, like retail applications, do not.

Example functionality:

-   -   Count people, measure people flow,    -   Measure gaze time,    -   Set event-based alerts.

4.2 Application

Deploying AWARE in retail advertising will enable new business models toemerge, such as but not limited to:

-   -   New customer revenue models.    -   Actual views/dwell metrics.    -   Location, campaign, day-part time metrics.

As an example, the interaction of a customer with a digital signage maybe analysed through a Spirit-enabled sensor. It may be possible toanalyse the image from the sensor and monitor for example the following:

-   -   Measure customer reaction in real time.    -   Rich pose/gaze/attention information.    -   Track customer from window to checkout.    -   Test different campaigns in real time.    -   Modify advertising in real time in response to gaze time.

Measuring gaze time may require more than just face detection. It mayalso require at least the following:

-   -   Tracking the position and pose of every person.    -   Monitoring their head position continuously.    -   Tracking peoples' behaviour both before and after the gaze.

AWARE is able to assess different levels of pose at varying distances asillustrated in FIG. 35. At short distance from the camera, AWARE maydetect head whereas at longer distance it may be able to detect fullfigure. It may also detect and measure the gaze time as well as thedwell time at display.

A proposed camera setup and associated region of interest is shown inFIG. 36 in which a camera has been mounted above the display. For thisparticular setup, the possible distances of detection areas for thedifferent poses are given:

-   -   Camera setup:        -   1920×1080 resolution        -   90° FoV        -   Mounted 2.5m from ground, tilted down at 20° below            horizontal    -   Detection areas:        -   “Head” detection:        -   X max, min 5.5m 1.0m        -   W max, min 8.0m 1.4m    -   “Upper body” detection:        -   X max, min 9.5m 1.0m        -   W max, min 13.5m 1.5m    -   “Full figure detection”        -   X max, min 17.5m 1.5m        -   W max, min 24.5m 2.5m

The detection areas are based on the assumed setup and choice of cameramodels. Actual results will also be influenced by the quality and typeof camera, lens and lighting conditions and may also be subject toocclusions.

Several implementations of AWARE are possible as shown in FIG. 37, suchas, embedded in IP camera (A), racked appliance (B) or PCI-server Card(C).

As explained earlier in Section 3.8, the limitations of security camerascan be overcome with ART and AWARE.

4.3 AWARE Analytics

Today, a small number of expensive cameras are usually installed innon-ideal location.

They often provide data that is of low quality and not very useful.Today's systems also lack from scalability.

One of the advantages of AWARE is that it is delivering a paradigm shiftin retail thanks to a network of smart sensors offering high qualitydata such that it makes it possible to capture shopper behaviour. Thisis mainly due to the fact that AWARE uses low-cost sensors, which areconvenient, and unobtrusive. They also require low bandwidthconnectivity, are simple to install and are fully scalable.

AWARE also provides real-time analytics on the behaviour of the detectedand tracked people. Some of the insights include for example, asillustrated in FIG. 38:

-   -   Keeping track of display awareness by measuring the number of        persons glancing by hour of day.    -   Tracking the percentage of passing population that observe a        shop display.    -   Measuring gaze time variation by analysing the gaze frequency by        gaze duration (people/seconds).

Within a shop environment, a deep analysis of customer behaviour ispossible as shown in FIG. 39. From the sensors located outside the shop(A), a vast amount of information may be extracted from measuring gazetime such as: Window display's efficiency in capturing consumers'interest, average gaze time of passing traffic, optimal time/day forcapturing consumer interest. From the sensors located inside the shopconsumer pathway (B) may be analysed, such as customers' real-timepathway through store ‘Entrance to Exit’. Hence in-store promotions maybe optimized in real-time, and the store regions' returns may beanalysed. In particular dwell time may be analysed in specific zones(C).

FIGS. 40 gives an example of a Web GUI for smart retail. The peoplecount for the day and the current people count are displayed along withthe day average gaze time. Charts on the average gaze time may also bedisplayed as well as statistics on the total people who returned to looka specific digital display. Plots of the average gaze time as a functionof time of day and of a count of people walking past as a function oftime of day are also shown.

The intelligent sensors enable real-time insight from consumer behaviourby detecting, tracking and analysing consumer behaviour. This can thenenable a tailored interaction in which enhanced services are presentedin real time and it is therefore possible to develop a high ratedpersonalised customer experience.

A marketing proposition is provided with sustainable personalisedproduct interaction, driving unassailable market leadership in digitalshowrooms. Premium service can be maintained for enhanced returns,driving high market growth and delivering high levels of consumer valueand premium service.

5. JSON-Based Event Generation and Event Switching

Method and device for JSON-based event generation and event switchingfor Smart Home, Smart Building and Smart City camera and sensornetworks.

5.1 Background

The smart home industry consists of selling sensor devices fordeployment in the home and controllers for using the data from thosesensors to control appliances or devices within the home. Thecontrollers can be software based and reside either inside a device orhub device within the home such as a cable provider hub box; or thesoftware can reside in a server in a datacentre connected through theInternet, a cloud-based controller.

There are three major problems with the practical integration of suchsmart home systems. The first problem is a commercial problem, which ishow to define the commercial boundary between the vendors of the sensorsand the vendors of the controllers and the vendors of the appliances.This becomes a particularly acute problem when there is more than onevendor in each of the 3 categories of vendor. For example, the homeownerbuys an energy monitoring system in the first year, then adds a securitysystem in the second year. However, both systems should preferably workoff the same camera sensors that the homeowner has already paid for. Tomake this work, the vendors have to agree a commercial boundary that canbe audited for transactions between the various systems to ensure thatvalue is shared and responsibility for value is clear.

In some instances, a single vendor will try to supply all 3 categoriesof product: the sensor, the controller and the appliance. However it ishighly unlikely that the same vendor will supply an energy, safety,security and entertainment home system all fully integrated.

Added to the above issue is the fact that some sensors are morefundamental than others. For example, a sensor that tells you who is inthe room is of value to many different controllers and appliances forthe smart home. These fundamental sensors should only be deployed oncein the home and the homeowner should not have to purchase new sets ofsensors for every different controllable system they require.

This patent addresses how the commercial interface problem would besolved where the interface is between a set of fundamental sensorproducts supplied by one vendor and a variety of system controllers suchas energy, safety, security and entertainment systems all supplied bydifferent vendors. The solution is to create an event subscriptionservice to which the system controllers would subscribe and thereforereceive event notifications and data from the sensor system.

The second problem facing the industry is the technical interfacebetween the various vendors. While standards are emerging slowly, thereis a huge range of products available in each category: sensors,controllers, and appliances. If the industry waits until every vendoragrees on every interface between every device it is highly unlikelythat a solution will be reached.

In the example given above with one vendor selling fundamental sensors(camera based) and four other vendors supplying control systems (energy,safety, security, entertainment), there could be multiple sensors eachtalking to every one of the 4 categories of systems. The amount ofcommunication chatter that would need to exist in an IP packet basedsystem would require a significant upgrade in the bandwidth capabilitiesof the home Wi-Fi network before it could support such a set up.

This patent centralises the output from the sensors into a hub, butdifferentiates the events created on a per service basis. This allowseach service to receive different data that is relevant to their servicefrom the group of sensors as a single intelligent sensor. This solvesthe technical interface problem as there is only one standards basedinterface into the hub but it is designed to be flexible enough torepresent a large variety of event types to cover the requirements ofmany different control systems.

For example, an energy control system might only be interested in roomoccupation in the home, whereas an entertainment system might beinterested in gesture based control of individual occupants. This patentsolves the problem of how to create a single technical interface to pushevents to both controller systems with significantly differentrequirements.

The most common user interface between homeowner and smart home systemsis via the web and using a web browser either on a mobile phone or acomputer. Therefore this patent chooses to use Javascript ObjectNotation OSON) to represent the events being created. This is astandards based notation specific to the programming language Javascriptand will allow a rapid development time of graphical user interfacesbased in browswers because the event data will be natively representedin the programming language of all browsers: Javascript.

Finally, the remaining problem is to be able to provide a quality ofservice guarantee associated with each service provided. For example, asafety service might be considered higher priority than an entertainmentservice. Therefore in the instance where the bandwidth on the home Wi-Fiand IP networks are reduced due to excess traffic or a fault developing,it is important that the home solution can differentiate the eventsbeing pushed to the control system by a class of service marker. Thispatent solves this final problem by creating a virtual output queuedevent switch.

While all three issues described above are illustrated in relation tothe application of the solution within a single home, there are furtherapplications that extend beyond a single home. For example, a largeenterprise building such as a multi-storey office, or a factory buildingcould also utilise such a solution because in either case there would bemultiple systems that need to operate in parallel such as security,safety, fire safety, energy supply and ventilation. The describedsolution is inherently scalable to larger scale buildings than the home.

Beyond single buildings, the described solution is inherently extendableto multiple buildings, such as within a campus or city, provided theevent generator and event switch location is placed at a centralisedconnection point. This point must be sufficiently close to the buildingsin question to avoid any delay issues for the control systems inresponding to events.

In both the larger building case and the city case, the solution solvesthe three problems described above, while also solving two furtherscaling issues. Firstly, as the number of control systems using theevents increases, the described solution can scale up while retainingsufficient level of performance to ensure that these systems work withinknown delay and quality of service boundary conditions. This solutionthereby supports any commercial arrangements in place with guaranteesrelated to operational uptime.

Secondly, as the number of sensors and events increases with theincreased scale of the deployment area, the network utilisation can beoptimised by the prioritisation of events and event responses. This isbecause a multi-level class of service regime can be practicallydeployed on events to mask out events of lower importance and therebyreducing the bandwidth usage of contested network resources in those keyand important instances where a response is required by security oremergency services.

Also, the event generator can significantly help in weeding out falsepositives, which are events that the sensor network initially flags asan important event worth noting, but that the event generator can use ahigher level of knowledge to recognise as a false alarm.

5.2 Statement of the Invention

The invention consists of a method to generate event objects from acollection of individual sensor inputs in which each event object alsocontains subscriber information and class of service. The sensors aretypically spread around a home, around a building, or around a city andare connected to the event generator using an IP wired or wirelessnetwork. The event objects are coded in JSON format so that they can bedirectly used in Javascript-based software on Browser User Interfaces(BUIs) and web servers, or easily interpreted by standard server sideprogramming languages or server Application Programming Interfaces(APIs).

The invention also consists of a device that queues the generated eventsand switches them into an output channel based on destination and classof service using a virtual output queueing system.

The combined method and device together enable controllers to subscribeto 3^(rd) party event generating systems in order to make theircontrollers more reliable and have greater functionality. This is turngives the end customer greater control and flexibility over many areas:how they run their smart control systems, which vendors they use, theability to change vendors over time, and it provides them with a moreefficient use of sensors that they've already purchased. It willultimately reduce the cost of each system through re-use of assets andoptimisation of energy usage when multiple different smart systems arein operation, which is also a benefit for the vendor.

5.3 Advantages of the Invention

The invention describes a clearly defined interface between the vendorof a control system such as an alarm system and the vendor of the sensornetwork feeding the control system such as a security camera vendor. Theinterface is based on a set of services that the sensor networkpublishes and the control system can subscribe to. The advantage of suchan approach is that the services can be monetised and audited by bothparties in a contractual arrangement in a pay-as-you-go manner or in amore traditional annual contract. Financial transactions can be based onservice subscriptions with a fully traceable audit trail and a clearbilling mechanism. The invention therefore enables commercialrelationships between multiple vendors to work in a practical manner.

Another advantage to the invention is the ability for the end user toswap out any vendor in a particular deployment scenario. The vendorstops their subscription to the service and the commercial billingsystem will respond immediately, even though the decommissioning of thephysical system may yet take some time. As soon as the servicesubscription is removed, the switching system will no longer queue theevents for that specific vendor, allowing a graceful degradation of theoverall system while a particular vendors equipment is removed orreplaced.

This also then provides an advantage to the end user who is not tiedinto any one vendor at either the sensor side or the control system sideof the overall architecture.

Another advantage to the invention is the speed with which a new servicecan be turned up. A new service can easily be added into the queueingsystem and event generation system and published to the control systemsfor utilisation. This will reduce the often lengthy time-to-market issuefor a completely automated sensor and control automated system, whereoften the integration work and setting up of billing can takesignificant project time in completing.

The approach of using an event switch to queue and differentiate servicehas a significant scaling advantage in that a quality of service can beretained as the system scales. The number of ports in the switch can beincreased without degradation of throughput or quality of service bytrading off latency provided the compute platform is powerful enough.The design can also be implemented in hardware, as it is a variation ona packet switch typically used within large Internet Routers.

By using a web services interface to the sensor and control networkdevices, the central event generator and switch can be housed in anylocation connected through the Internet to the sensor systems or controlsystems. The advantage of such an approach is that the event generatorand switch can be deployed on a Content Distribution Network spreadingthe work across multiple servers.

The internal interfaces within the event generator and switch system aredefined sufficiently to split the internal processes and deploy themwith internal web services interfaces between multiple servers. Thisprovides an advantage in terms of scaling beyond a single server orphysical device to a set of racked servers where each serverconcentrates its workload on one aspect of the overall system.

Another advantage of the invention is the modular approach and design inthe separation of sensor output, rule generation for events, eventgeneration itself, service publication, billing and subscription, andevent priority queueing into individual systems. This is an advantagebecause the system is the meeting point of many different systems,vendors and end users in the scenarios described above and when one ofthese parts needs to change or be upgraded, it should not involve adisruption to the entire system which could have real-time requirementsfor a high degree of “up-time”. This will be a particularly acuterequirement for security and health monitoring systems, emergency andfire safety systems and other applications such as traffic control whereany disruption may cause accidents and health threats to people in thevicinity.

The use of JSON as the core unit of event generation and event queueinghas several advantages over existing systems. Normally the eventgeneration is achieved in a proprietary manner in a single vendor'sclosed system, however by using a standard data interchange format suchas JSON, it allows multiple vendors to participate in the eventgeneration and usage. While XML could also achieve similar results, XMLcan often be lengthier in output data size, and as the number of eventsscales up, it would be a less efficient usage of compute and networkresources than JSON. JSON is also a native data interchange format forweb programming such as Javascript on the client side and Node.js on theserver side, but can also be easily interpreted and exchanged betweenany server side programming language and distributed over an IP networkin a standard manner.

Switching would normally be achieved using an IP packet based switch oran Ethernet frame based switch which would switch packets at thetransport layer (layer 4 or below in the OSI network model). Theinvention has the advantage of switching at the application layer makingit applicable no matter what the underlying transport switch technologyis deployed. This means that the guarantees around the behaviour ofquality of service and the deployment of the services do not rely on howa particular switch vendor has implemented their switch. While the lowerlevel packet switches are still required, they are not involved in theswitching of the application layer directly. This allows the performanceof the application level switch to be changed, improved, developed,evolved and designed independently to the packet level switches,removing a potentially complex interdependency on performance levels.Ultimately, this advantage will enable a more widely deployable systemwhile retaining a similar level of performance across multiple networkdesigns and implementation scenarios.

The invention uses a rule-based system to generate events that utilisesa set of models of the physical buildings and individual rooms involvedas well as a set of rules that are both time based and multi-sensorbased. These rules are communicated between a centralised web server andthe various deployments using JSON formatting also and a web servicesinterface to control the exchange. The advantage of this approach is toallow the models and rules to evolve over time independently to theevolution of the sensor layout.

There is an advantage to using an open model and a rule-based system fortriggering events and relating them to the environment. The sensordeployment is published and the physical location of the sensors can beoverlaid on the physical model using JSON format so that rapid creationof web-based user interfaces can be created on-the-fly to communicatethe meaning of an event to the end user.

For example, the first vendor to deploy sensors can build a model of thephysical environment and publish it on the system. Subsequent vendorscan use this available model or combine previous sensor data with theirown to offer a new set of event rules that overlay on the same physicalmodel.

Equally this has the advantage of giving control to the end user of therules for event generation and the ability to control the detailsprovided in the physical model, while still allowing developers to testa potential new addition using the web-base published rules for thatdeployment before installation.

5.4 Description of the Drawings

The first sets of drawings [Figures 41-44] illustrate the invention inthe context of a smart home environment. The second sets of drawings[Figures 45-47] illustrate detailed implementation architecture of theinvention. The next sets of drawings [Figures 48-50] illustrate examplesof the various JSON data formats that the system uses. The next sets ofdrawings [Figures 51-54] illustrate the application of the invention tolarger scale scenarios beyond the scale of a single home environment.The next sets of drawings [Figures 55-59] illustrate the detailedimplementation of the invention with examples of data formats,algorithms, real environment data and explanations of how a deployedsystem would operate.

FIG. 41: Smart home architecture with hub software—this is a diagram ofa home with a single room illustrated [102] that contains 2 sensors[103] and an occupier [104]. The 2 sensors are connected eitherwirelessly or by cable to the home IP networks [105], which is astandard IP network hubbed around a single IP router that is used toconnect the home to the Internet. There is a device called a Home HubHardware Device [106] connected to the hub, again either wirelessly orby cable and this could be a standalone device, or could represent aset-top box for cable TV, or a gaming device such as the MicrosoftX-Box, or other similar home device such as a computer connected thehome IP network. Inside this hardware device is the “Hub SoftwareProgramme” which is the embodiment of the invention [107].

FIG. 42: Internet connected smart home with more rooms and occupantsshown—this is an extension of FIG. 42 with similar components that arenot described again, such as the home, room 1 and occupant, the home IPnetwork and the home hub hardware and software. This diagram illustratesthat the home is connected to the internet [201, 202] which provides analternative location for the “Hub Software Programme” which is thesubject of this invention. In this diagram, the Hub Software Programme[204] is located on a server running as a cloud server connected to theInternet [201] such as a server located within an Amazon data centre.There is also some network attached storage [205] also located somewherein the cloud infrastructure that can be used to store sensor data foraudit or review in the future. For illustration purposes of anembodiment of the invention, a second room is added to the home [207]that contains 3 more sensors [206] marked numbers 3, 4 & 5. In bothFIGS. 41 and 42, the sensors in the rooms are feeding raw sensor databack to the Hub Software Programme continuously where it is beingprocessed as described herein.

FIG. 43: Smart home with energy control system—this diagram is anextension of FIGS. 41 and 42 and for illustration purposes only, itshows Room 2 in another context on the right hand side of the diagram[305]. Note that this is not a 3^(rd) room in the house, rather itillustrates the other side of the smart home equation, which is the useto which the processed sensor data is being put. In the scenarioillustrated here, the home occupants marked #2 and #3 [306] are in room2, which has some energy appliances marked #2 and #3. There is now anenergy controller, which could be both a hardware device and software orany combination therefor [304]. It could be located as shown in the samehardware device as the Hub Software Programme or it could be located ina different hardware device inside the home. The diagram also shows thepossibility of the energy controller being located on a cloud serverattached to the Internet. The sensors send raw data to the Hub SoftwareProgramme where it is processed and events are created. These events arepushed across the network to the energy controller that has directcontrol over the energy appliances shown in room 2.

This system allows the energy controller to recognise that occupants #2and #3 are in the room and therefore adjust the energy system to providea desired temperature for those occupants as per their own personalsettings, while also optimising other parameters such as the overallenergy bill of the household. The workings of the energy controller,preferences of the occupiers in this regard and the energy appliancesare for illustration purposes and this invention does not relate to thetechnology contained in those devices or methods. There is a clearinterface between the invention disclosed and such systems, which is thepushing of an event in the form of a JSON-based data interchange betweenthe Hub Software Programme and the energy controller through an IPnetwork.

FIG. 44: Smart home with 4 controller systems: energy, safety, securityand entertainment—this diagram is an extension of FIG. 43, wherebyseveral other system controllers are illustrated such as anEntertainment controller [401], a Security controller

and a home Safety controller[403] located in cloud servers connected tothe home via the internet. Each of these controllers behaves in the sameway as the energy controller described in FIG. 43 above. They all belongto 3^(rd) party service providers and they each receive a set of eventspushed to them from the Hub Software Programme. Each individual systemmay have its own controllers located within the home also such as shown[405, 406, 407]. Each of the controllers, whether in the cloud or in thehome itself may have devices that they can control as illustrated by[409, 410, 411] shown in Room #1 [408]. Overall note that the scope ofthis patent is limited to the Hub Software Programme [404] locatedeither in the home on a suitable hardware device or in the cloud on asuitable server connected via the internet to the home IP network.

FIG. 45: Hub Software Programme architecture and components—This diagramillustrates the main architectural components of the Hub SoftwareProgramme [501] and relates directly to the claims of the patent. Thereare 2 major parts to the Hub Software which are the Event Generatormodule [502] and the Event Switch module [503].

The Event Generator module shown on the left hand side then has 4components to it, which are (counter clockwise from top left) aManagement Module [505], A Sensor Data Buffer [507], an Event Generatormodule [509] and a Home Model & Event Rules module [508]. Thesecomponents are dealt with separately in FIG. 46 below, but in generalthey combine to produce individual events which are human recognisableactions such as leaving a room and entering another room, built up fromthe combination of multiple sensor outputs (shown as Sensor Data Input[506]) combined over specific time periods. Each event generated by theEvent Generator module [502] has a unique rule associated with it thatis used to calculate the event and all events relate to one or moreservices that provide the events to the user via a web servicesconnection model.

The Event Switch module shown on the right hand side then also has 4components to it, which are (counter clockwise from top left) aScheduler Module [512], a Virtual

Output Queue module [517], an Output Buffer module [516], and aManagement Module [514]. Events are passed to the Event Switch from theEvent Generator module [511], where they are queued according to theirsubscriber and their class of service. This means that a subscriber suchas a provider of home Energy solutions, can subscribe or ask for certainevents to be sent to it [515]. Each event type will be given a class ofservice so that more important events can be given priority in the eventof multiple events existing in the queues that are competing for limitedoutput resources on the IP network. For example, a safety relatedservice may be given higher priority than an entertainment relatedservice. Each class of service shall have a known latency and guaranteedlevel of service (e.g. reserved bandwidth capability) in order to createa commercially sound interface between service provider and servicesubscriber, one on which 3^(rd) party engineering systems may be builtto sufficient robustness to charge the homeowner for a level ofguaranteed service.

Note that the management modules [505][514] communicate with amanagement server located in the cloud and connected via the Internet tothe Hub Software Programme host hardware [504] [513]. These are separatecommunications channels logically separated from the dataflow through aseparate web services interface. These channels are used to pass updatedsets of rules, updated subscriber information and updated models of thehouse to the Hub Software Programme from the management system based ina data centre cloud operating system.

FIG. 46: Event Generator Module architecture and components—this is amore detailed view of the Event Generator Module on the left hand sideof the overall Hub Software Programme illustrated above in FIG. 45. TheEvent Generator Module [601] has 4 component modules, 3 of which aredescribed in this diagram. The 4^(th) module, the management module isnot described in more detail as it consists mainly of a means tocommunicate persistent data to a management server and receive updatesof the various operational configurations required for the other 3modules.

Starting at the top right and moving around clockwise, the first moduleis the Home Model and Events Rule module [602]. This contains two setsof data, the Home Model set of data [603] and the Event Rule set of data[607]. The Home Model set of data is a simplified model of the home[604], of the room connectivity [605] and of each individual room [606],which contains information about the size, shape, content and make up ofeach room in the home in a format that the rules for event creation canuse to determine desired event occurrences such as movement of occupantsbetween rooms. Each Service such as Service #1 shown here [608] containsa set of rules for event calculation. These rules take the sensor datafrom multiple sensors and combine them in an algorithmic fashion over adefined time period to ascertain patterns and create events from thosepatterns, such as the movement of an occupant from one room to anothermay be an event service that an energy controller system could subscribeto. The rules are stored in a Rule Table [609], and there is a RuleTable per Service.

On the bottom right is the Event Generator [613]. This comprises of aRule Sequencer [614] and a set of Rule Processing Blocks [615]. The RuleSequencer creates an ordered list of when each rule is to be calculatedso that it can meet its particular service performance levels. The RuleProcessing Blocks take each rule and break it down into a series ofcalculations that are performed in sequence. There are multiple blocksto allow a “manufacturing line” sequencing of rule calculations in orderto optimise processing time. The output [616] is an Event formatted inJavascript Object Notation (JSON), which is then sent to the EventSwitch described further above.

On the bottom left is a Sensor Data Buffer [610] which receives blocksof sensor data [612], e.g. in the form of XML files, and queues thembased on the sensor they came from. This queue is essentially a 2-Dmatrix of data for each sensor [611] over a time period equal to thelength of the available memory buffer. The buffer should be long enoughto hold a sufficient amount of incoming sensor data required tocalculate the various service rules, given their expected timeframe,e.g. moving from one room to another is an event that could take up to20 seconds, so the buffer should be capable of holding 20 seconds worthof data.

FIG. 47: Event switch module architecture and components—this is adetailed view of the second major component of the Hub SoftwareProgramme which is the Event Switch Module [701]. As mentioned whendescribing the first major component, there is a management moduleindicated by the top right segment of the component [701] which performsdata persistence functions and Operations, Admin and Maintenance (OAM)functions that are not detailed here. It also allows schedulingalgorithms within this block to be updated or modified. The top leftsegment is the scheduler indicated in a previous diagram and more detailis given in a diagram further below.

This diagram then focuses on the bottom left and bottom right segmentsof the component [701]. The bottom left is the set of Virtual OutputQueues [702], which receives Events in JSON format from the EventGenerator and enqueues them

[709] into a Virtual Output Queued buffer in preparation for schedulingand switching the Events to an Output FIFO buffer [710]. It achievesthis by using class of service to dequeue the events [704] in aparticular order determined by the scheduler. In this diagram, there are4 sets of queues, one for each of 4 current services [706] [707] [708].Within each of these service sets the events are then queued based ontheir class of service. So effectively events are queued based on theirdestination which is the subscriber who has subscribed to an eventservice, and within that destination the sequence of events being sentis determined by the class of service. Given that there may be manysubscribers and only one or two network output channels to transmit theevents, this must necessarily switch the queued events into those outputchannels based on the scheduler rules applied and the resources e.g.bandwidth available.

FIG. 48: A house model using JSON notation—this diagram is a codingexample of a JSON-based data model for the smart home. The home is givena name first, and then there is an array of room objects with variousparameters described. Each room is given a name, a shape, anorientation. Then the walls, floors and ceilings are given single ormultiple levels and their dimensions are described which also thenprovides the overall dimensions and volume of the room implicitly.Doorways, windows, objects and lights are also provided in this example,but the model is inherently extensible to many other parameters that maybe required to produce intelligent events related to the physical homeitself. Finally, an array of connections for each room provides themodel with the ability to map out how each room connects to each otherroom in the home. This model is sent using JSON directly to themanagement module of the Event Generation module and can be updated asthe home changes at any time using an online management interface hostedin a cloud based server accessed through the internet. Dimensions shownas either integers or decimal numbers are all given in meters in thisexample. Specific header information may be included that states theunits of measurement.

FIG. 49: An Event in JSON Notation—this figure is a code snippetillustrating a JSON-based data format for an event in the home. In thiscase the event is an occupant changing room and the service is called“Presence-A”. This event generation service has class of service 2 as itis not considered a safety or security hazard as it is based on thenormal day-to-day movement of a recognised occupant within the home. Inthis instance, the people that this event reports on are “homeowner” and“child1”. Note that the people do not have to be named for privacyreasons for the system to operate successfully. In fact the system neverneeds to know the occupants names. Although in this instance, theoccupants are given relevant titles to distinguish them from each other.

The recognition would typically be done using a snapshot of a suitableangle of the face which is then sent to 3^(rd) party facial recognitionsoftware.

The event also lists the subscribers to that event, information whichallows it to be queued correctly in the virtual output queue within theevent switch module. In this instance, 5 subscribers are shown withtheir brand name and their service type that they provide to theoccupants of the smart home. The event shows the start and end times andthe start and end positions of two occupants moving from the hall to thekitchen. Note how the event is readable and contains very littletechnical detail so that it can be easily interpreted by many differentservice designers and user interface designers.

FIG. 50: An Event Rule in JSON Notation—this diagram shows a codesnippet of a JSON-based data rule for generating events. Each servicecan have many different rules and as services and added or removed, thelist of event rules can be updated through the management module of theEvent Generation module using a web based management interface. In thisinstance, the rule is described as a “Room Change Rule” which simplygenerates an event when specific occupants move between rooms.

Initially the rule is given a unique ID, the name of the service itpertains to and a class of service level (2). Then the subscribers tothe service are listed. This helps the Event Generator to schedule therule calculations so that the correct rules are calculated for thecorrect time slots. It is also how an Event that is generated is givenits subscriber list so that it can be queued later in the outputprocess.

There is an include and an ignore people list, so that certain occupantsmovements are either included in the rule calculation or note. Thesensors involved are listed, and finally a time window is given. Ofcourse many more sophisticated rules can be generated in this manner. Ahuman readable description of the rule is provided at the end for userinterface purposes when the event is pushed to the subscribing systemsand they involve user interfaces, such as an energy control panel, orthe TV screen of an entertainment system.

FIG. 51: Application to an Enterprise Single Building—the invention canalso be applied to larger scale buildings beyond the size of a singlehome. In this diagram, the solution is illustrated for an EnterpriseBuilding [1110] with 12 offices [1114] and a server room [1111]. Thebuilding has its own internal IP network [1108], which is connected tothe Internet [1104] via a business Internet connection [1105]. There are2 sensors shown per room and 1 occupant [1114], and all sensors areconnected to the internal IP network [1108] via a 10/100/1000 EthernetLAN [1113].

In this scenario, the Hub Software Programme [1109], which is the scopeof this patent, is located either on a server [1112] in the server roomwithin the building [1111], or in a cloud server with networked storageconnected to the building via the Internet [1101][1102][1103].

A single subscribed service is illustrated as an Energy Controller[1106] [1107] located in the cloud which receives events via theinternet and can control the energy systems in the building remotely.Note that the Energy Controller could of course reside somewhere in thebuilding too if required. The invention works exactly the same way nomatter where the Hub Software Programme is located or where the EnergyController subscribing to the service is located as long as they areconnected via an IP network to each other and to the sensors involved.

FIG. 52: Application to an Enterprise Campus of Buildings—this diagramillustrates the application of the invention to an Enterprise Campus ofBuildings which consists here of a Main Campus Building [1214], and a2^(nd) smaller campus building [1215] and a 3rd small campus building[1216]. Each building has various rooms with sensors and occupants insimilar fashion to the smart home and smart building mentioned above.Each building also has its own Building IP Networks [1208], and all 3buildings are then interconnected with an IP Campus Networks [1217].There is a business internet connection [1205] connecting all buildingsto the internet [1204] via the campus wide network [1217].

The Hub Programme Software is either located in one of the buildingswhich is large enough to have its own server room [1209][1211][1212], orit is located in the cloud [1201][1202][1203]. A campus energycontroller also located in the cloud is shown [1206][1207], but asbefore both the Hub Programme Software and Energy Controller can besituated in either the cloud or somewhere on the campus as long as theyare connected together via the IP network or the Internet, and connectedto the sensors [1213].

FIG. 53: Application to a City Block—this diagram illustrates theapplication of the patent to a City Block (or street) with 5 buildings[1304][1305][1306][1307][1308]. The last 3 buildings also have Forecourtareas or Lots which also have external sensors or cameras [1313]distributed throughout [1309][1310][1311].

All of the same application ideas apply to this street scenario as withthe other scenarios described above as long as all the components arenetworked together. For external cameras/sensors this may in factinvolve a mobile (wireless) network connection through to the Internetand cloud based servers involved.

FIG. 54: Simultaneous Application to a Smart Home, Enterprise Campus andCity Block—this final application diagram illustrates how the sameinvention can be applied to all of the above scenarios simultaneously bybanking together multiple instances of the Hub Software Programme in thecloud [1406]. The diagram show a smart home, smart enterprise building,smart enterprise campus of buildings and a street or city block scenario[1401][1402][1403][1404] all connected back through the internet to abank of servers hosting the Hub Software Programmes for each, and alsothen connected to a bank of suitable controllers [1408].

Note that it is possible to add a Master Server which can act as a hubfor the multiple hubs and controllers, and there is a second such MasterServer shown for redundancy [1407]. These Master Servers may do littlemore than aggregate the events from each Event Generator and create asingle Event Switch, or they may also act as a second tier of multipleEvent Generators and Event Switches that sits above the initial bank andprovides a set of global events from all the attached locations into thesubscriber services and controllers.

FIG. 55: Raw Data From Sensor—this diagram gives an example of the rawdata that is produced by a smart camera sensor for tracking people andobject movement and position in the various scenarios described. On theleft hand side is a structured standardised XML document architecture[1501] describing how the raw data from each sensor can be packaged upfor transmission back to the Hub Software Programme. When it arrivesthere, it can be parsed and then used for calculations in the EventGeneration module.

There is a header first with configuration information, followed by alist of sensor identifiers in case there is more than 1 sensor involvedat that location and all sensors can add their data into the same XMLfile for efficient transmission back the Hub Software Programme. Next inthe XML file is a list of objects that have been discovered, some ofwhich are placed into intelligent groups (e.g. body, head and shouldersdetections might be combined to form a group representing a person).Finally the groups can be connected in time from one frame to the nextvia the Track grouping method.

On the right hand side is an example of a real XML sensor data filetaken from a deployment at an electronics store. This exampleillustrates how the information for each grouping is bundled into a<set> element, and there is detailed information about each object suchas the attributes of its size, parameters used during detection, itscoordinates, its angle (pitch, roll and yawl) various identifiers.

While XML is well suited to the engineering level data produced by eachsensors due to its structure and extensibility, the events generatedhave to end up with human readable form and be used in client-serverarchitectures with the smallest size possible, and so the events arethen generated in JSON format as described above rather than XML.

FIG. 56: Sensor Object, Group and Track Examples—this diagramillustrates the real world tracking data that a smart sensor producesand which will feed into the Hub Software Programme. The examples aretaken from smart camera sensors produced by Apical using machine visiontechniques to identify people [1601][1602][1603] and then representingthem as “ghost” forms with metadata [1604], thus retaining the privacyof the individual while allowing real world information to be passed tothe Event Generator for the Hub Software Programme to build a usefulservice.

FIG. 57: Rule Implementation—this diagram gives and example of how asimple rule can be implemented for using the raw sensor data from twosensors in adjoining rooms and working out if an occupant has movedbetween the rooms. On the left hand side are the 2 rooms [1701][1702],both with a camera sensor [1704][1705] and the rooms are connected viaone doorway [1703]. Initially the occupant is in Room 1 but is trackedmoving towards the door [1706]. A new track is created in Room 2 [1707]when the occupant enters that room and comes into view of the camerainside.

The right hand side of the diagram shows a Pseudo-Code illustration of arule using a flow diagram for the Pseudo-Code logic. This simpleRoom-Change Rule tries to match the tracks in both rooms over a timeperiod to see if indeed an occupant in one room has moved to anotherroom. There is a confidence level then associated with the result whichis increased if facial recognition algorithms have matched the occupantin Room 1 to the occupant in Room 2 and assessed that they are indeedthe same person who has changed rooms.

FIG. 58: Rule Sequencing—this diagram illustrates an important part ofthe method of the invention which is the Rule Sequencer. There will be acollection of rules related to each individual event service providedand even some rules specifically implemented for individual subscribersolutions that are not available to other subscribers. There is a finitecompute resource available within the host system for the Hub SoftwareProgramme which should be optimised, as it may reside on an existinghome hardware device that does not have a lot of available memory orcompute power to offer. Therefore a Rule Sequencer, examines the rulesand the resources available to compute them, and creates a sequence forcalculating the rule for optimised performance. The Rule Sequencer isdesigned to make sure that the rules are calculated in time to createevents that meet the service level agreement with the subscriber. Forexample, an event might need to be produced every 3 seconds as shownabove. The Rule sequencer merges rule calculations with other rules insufficient regularity in order to meet this service requirement. Notethat the granularity of the rule calculation sequence in terms of timeperiods should be much shorter than the granularity with which theservice events and corresponding rules are required to be computed. Thiswill allow room for the Rule Sequencer to manoeuver rule calculationsinto a sequence that can meet all service demands on time. For example,the events could require a granularity of 1 second each. The rulesequencer would use a granularity of 0.1 seconds for each rulecalculation in sequence.

FIG. 59: Scheduling—this diagram illustrates how the scheduler workswith the Virtual Output Queueing system of JSON-based events [1901] inorder to schedule transmission time slots filled with suitable events tomeet the service level agreements with the subscribing systems. Theevents are queued up [1902] before the scheduler and switch to outputports, this is therefore known as an “input-queued switch” architecture.Events are queued first by destination—which in this case is thesubscriber system, then within that by class of service of the variousevents that that subscriber has requested. For illustration, 3 classesof service are shown [1901]. Each queue is populated with a small numberof events to demonstrate how the scheduling system operates.

In this scenario, there is only one output port, but the scheduler couldequally handle multiple output ports and schedule time slots for eachport. The scheduler's task is to fill a time slot with events such thatthe service level agreement around a class of service is fulfilled. Forexample, each timeslot will use up 75% of available resources for classof service 1, then the remainder will be given to the other two classesof service based on the level of the event. A class of service level 2will always be given the 25% resource remaining ahead of class ofservice level 3 for example.

The scheduler can look across several events within each queue in orderto make a decision, e.g. using a window of 2 events in this example.Timeslots [1905][1906][1907][1908] are then created one after each otherand filled by the scheduler with events for transmission. The timeslotis there to help the scheduler meet its requirements but the events arethen transmitted one at a time starting with the first event in eachtimeslot, when transmission bandwidth on the IP network becomesavailable. Here, the first time slot is 75% filled with COS1 events,then the remaining 25% is filled with COS2 events, both taken from thefirst 2 events in each of the queues. The second timeslot starts to fillup with COS1, but then there are no more COS1 events to fill it. So itthen moves to COS2 events until they too are used up. At this point thescheduler is only looking at events 2-deep in each queue, and so it usesup all available COS2 events. It therefore moves to COS3 to fill theremainder of timeslot 2. In timeslot 3, the scheduler now looks deeperinto the queue and finds more COS2 events to start filling the timeslot.Note that there are is only one more COS1 in any of the queues, so theCOS1 event is placed first then the scheduler moves to COS2 events.Finally timeslot 4 mops up the remaining COS3 events in the queue.

Note that more events would be then joining the queueing system and thescheduler would continue to work, but this example is bounded in time inorder to show a complete emptying process for the queueing system. Thereare many ways to build such scheduling algorithms and this is oneexample of a simple resource allocation algorithm based on a round-robinof each queue per class of service.

5.5 Detailed Description

Assuming that the context of the invention, the problems it solves andits advantages are described above, this is a detailed description ofhow the Hub Software Programme first illustrated in FIG. 41 [101]operates. This is the embodiment of the invention and consists of amethod and a device combined. The method is a means to generate eventsand supply them as a subscribed service, the device is a means to switchthe events for distribution to meet a quality of service guarantee.

The method to generate events and supply them as a subscribed service isembodied in an Event Generation module shown in FIG. 46. The operationof this module begins with the arrival of raw sensor metadata that istransmitted over the home IP network from the camera or sensor to thedevice that houses the event generation software. This raw sensor datais queued in a buffer which is typically created as a 2-D array of data,with one dimension representing time and a second dimension representingeach individual sensors that has transmitted data to the hub.

Note that often the individual sensor metadata may be out ofsynchronisation with the metadata from other sensors and therefore theremay be a block of processing required to normalise the metadata withinthe buffer. This could be implemented as a second buffer that replicatesthe structure of the initial buffer, and the normalised metadata wouldappear in the second buffer. The normalised data is now ready to haverules applied to it in order to create events.

The event generator module also contains a block of software functionsto manage the collection of home models and rules from some centralisedserver based in the cloud and connected to the hub via the home Internetconnection. These models and rules are quasi-static in nature and areoccasionally synchronised with the cloud server to ensure consistency.The home model contains data about the structure and content of therooms of the home, and the event rules contain calculation instructionsfor how to produce an event.

The rule sequencer is a block of software that reads in the home modeland event rules and creates a sequenced implementation of their use incalculating events. The calculations are applied to raw sensor metadatastored in the buffer and the output is an event which is a small blockof data containing information about the occupant(s) of the areas of thehome that the rules are applied to. Examples of events are presence,movement, unusual behaviour, gestures etc.

Each event calculation is inherently part of a service that the softwareoffers, such as a presence service. Therefore each event is marked withthe appropriate service reference, which could be simply a service name.It is also marked with a user ID, which represents the customer orcontrol system that is going to utilised the constructed event.

Likewise, each event is being calculated for a customer who has appliedfor that service, and the customer has signed up to a Service LevelAgreement which translates into a quality of service for that supply ofevent data. Therefore the quality of service value is also attached tothe outgoing event data from this module.

So now the event generation module is outputting blocks of data calledevents which contain pertinent information about a particular event.Each event is marked with a service identification number and a customerID, and also with a quality of service. This event is then passed ontothe second major block of the invention: the device that can switchflows of events in order to handle network resource conflicts and ensurethat each service adheres to its prescribed quality of service level asagreed with the end customer.

The switch device is illustrated in FIG. 47. It is a virtual outputqueued switch, which means that it forms queues at the input side of theswitch that represent the destination of the data on the output side, ora virtual representation of the output queues that would exist. Giventhe event data queued in such a manner it is possible to build a switchscheduler that controls the switch fabric resources sufficiently toensure that each event gets switched through the fabric and into theoutput buffer in time to meet its Service Level Agreement.

Each event that arrives is therefore queued based on the end userdestination which is the entity ID that subscribed for the event, andthen within that queue there is a series of internal queues for eachclass of service. So for example if there were 3 customers and 3 classesof service, there would be 9 queues in the Virtual Output Queue.

The dequeue function works from a set of rules contained in thescheduler block which is shown in FIG. 45 [512]. This scheduler ismaking the decision on which of the Virtual Output Queues to empty nextand in what proportion of time and filtering the dequeued elements intoa Output Buffer [516]. The Output Buffer is a FIFO (First-in-first-out)single queue which then sends each event in the sequence it was queuedin to the address of the subscribing customer contained as a dataelement within each event. This Output Buffer does not need tounderstand anything else about the data other than to send the nextevent in the queue to the address by which it is labelled.

Both the Event Generator and Event Switch modules have their ownmanagement functions that help report errors and warnings up to acentralised management system that would sit on a server in the cloud,and communicate with the hub software via the home Internet connection.Updates to either module can then occur separately, including potentialsoftware updates to enhance the performance of the system etc, withoutinterfering with an existing version of the module that is not beingupdated. These management functions are shown in FIG. 45 [505] [514].

The management functions also monitor the communications channels forpotential errors and they gather statistics on the performance of themodules for use in auditing for billing issues for example. Themanagement modules may also be used for performing internal monitoringfunctions such as the direct transmission of raw sensor metadata up to anon-customer cloud server, by-passing all of the main functionality ofthe event generation or event switching. This data would then be used tohelp further develop the research and development programmes for theproduct to improve performance, reduce power consumption, feedback intonew scheduling designs or improve any aspect of the technicalimplementation or algorithms of the system.

The above description of the operation of the event generation and eventswitching modules also applies directly to the application areasillustrated in FIGS. 51, 52, 53 and 54. There would be no change to theoperation of the modules other than the amount of data that each modulewould need to process. However the architectures described areinherently scalable as for example the scheduling algorithms of avirtual output queued switch are generally scalable to hundreds of portsas proved by their usage in the switching fabrics of large scaleInternet Routers such as Cisco CRS1.

Examples of the data structures and algorithms used are given in FIGS.48, 49, 50, 55, 57, 58 and 59. These illustrate one possibleimplementation of each of the important data components of the system ordata processing algorithms of the system. In particular, the JSONexamples illustrate how the data can be organised into logical blocks asassociated arrays ready for conversion directly into coded objects foruse within Javascript based browser user interfaces. This should makethe combination of the metadata of an event with a visual image or videomuch more straightforward for a 3^(rd) party web programmer who has torapidly and continually adapt the user interface based on the home modeland users involved in each specific deployment. This degree of freedomwill be facilitated by having the data ready packaged in JSON notationfor the javascript animations or HTMLS modern elements of browser baseduser applications.

The examples of algorithms are basic scheduling algorithms such as thecreation of time slots shown in FIG. 59. This example could be replacedby any of a large collection of known methods to implement a schedulingalgorithm for queue selection such as a round robin, random,pseudo-random selection etc.

The JSON-based models of the home and the rooms for example are designedso that a browser user interface or any web application could be quicklyand easily adapted to allow the end user to build their specific homemodel. A range of rapidly changing user interfaces may need to becreated to facilitate different methods of extracting the pertinentinformation about the users' home and creating a suitable model for usein the event generation. Note that by formalising the models, it alsoallows for machine-to-machine communication of known or previouslygenerated models so that the user could simply select a model that isclose to their own, which could have been generated by a neightbour forexample.

5.6 Protocol Example

This section outlines and walks through an architecture and approach forART that will allow Apical to engage commercially with partners with awell defined technology and commercial interface.

a) Lower Level Sensor System & Apical Hub Software

Every camera or sensor that contains an Apical FPGA sends a constantstream of XML formatted raw data to the home hub.

In the home hub is an Apical software application. The Apical softwareapplication takes the raw sensor data from multiple sensors andautomatically creates various events as follows:

Example with 3 events:

-   -   Event type 1 takes a single input and immediately outputs a        presence event with a 70% accuracy.    -   Event type 2 takes in multiple inputs and uses a window of 2        seconds to output a presence event with a 95% accuracy.    -   Event type 3 takes in multiple inputs and uses a window of 10        seconds to output a motion event with a 99% accuracy.

The software computes these events automatically and queues thedifferent events into their own queues.

There is one queue per event type.

An event type may be defined by 4 criteria:

-   -   The event type    -   The number of sensors it uses    -   The window or history length is operates on    -   The confidence level

Events are queued as they occur in a FIFO queueing system, which timestamps the events upon entry, and keeps a stack of time spent in queuefor the events in each queue.

b) Higher Level Apical Hub and Apical Cloud Software

All of the above is happening constantly in the background.

The Apical cloud software can talk to the Apical hub software and set aregular time interval after which all of the data gathered is formattedinto a file and sent using FTP to the Apical cloud software. This datais for Apical to use for legal reasons and for maintenance, debuggingand development reasons.

Note the Apical cloud software can also request the Apical hub toconcatenate the raw data XML streams into this file and send both theraw data and the event data up to the cloud software for analysis.

This could be very useful for debugging problems or issues. Meanwhile,there are several higher level smart home management systems inexistence. For example, a system for home entertainment, and anothersystem for home energy management. Each of these higher level systemscan subscribe to an Apical service by using the Apical API based on aRestful Web Services interface. This subscription process is handled inthe Apical cloud software. The management system server retains a recordof all subscriptions for billing purposes. The Apical cloud softwaretells the Apical hub software which higher level systems have subscribedto which services. The Apical cloud software provides the local IPaddress of the hub of the higher level management system that hassubscribed to the service. The Apical hub software negotiates an IPunicast stream to that IP address.

A unique set of queues is set up that contains all of the events thatthe higher level system has subscribed to.

c) Class of Service

There is one queue for premium services and one queue for best effortservices.

The premium queue takes precedence when there is a queueing conflict fortransmission, effectively offering a premium level of service.

There is a known latency associated with the queueing system, which isessentially a length of time beyond which no guarantees that an event,which has been queued for that length of time, will actually betransmitted.

The best effort queue has a fixed length (e.g. 2 seconds worth of data)and events arriving later are dropped from the queueing system if thebuffer is full.

The Apical hub software can report to the subscriber that they havecurrently subscribed to too many services and must either drop a serviceor promote a service to premium level if it is to be serviced correctly.

The Apical hub software should be aware of the approximate bandwidthavailable on the home wireless network being used.

The underlying protocol or type of home network is irrelevant, so longas an TCP/IP connection or a UDP connection can be established andmaintained.

This may work on IEEE 802.11xxx, or on any new type of home wirelessnetwork provided they can support devices with IP protocol stacksrunning. This will almost certainly be the case.

d) Higher Level Services (Apical Partners)

Each higher level system that subscribes to the Apical service is billedmonthly or annually for the service.

Note there are 3 reasons for the use of a constant stream where data ispushed to the hub and then pushed from the hub to a higher level system(such as a home energy management system):

-   -   1. It will be necessary to size the system (bandwidth, switching        throughput etc) for the subscribed services anyhow, the optimum        engineering solution is to send the event object when available        and let the higher level systems put aside or get rid of the        ones they don't use. The reason is that you are in effect        running a continuous check that the link is up and running and        therefore your management system and your partners ecosystem has        a pulse that tells it that everything is ok. If you stop a pulse        being sent through, you have no way of knowing that the system        is live—it acts as a cross check.    -   2. A secondary advantage of a push system versus a pull system,        is that the level of 2-way communication that must occur to        request a “pull” of data off the sensors or off the Apical hub        is far more complex (and therefore prone to errors) than a        simple push model. The data is pushed and then the receiving        system can queue it, dump it or use it, Apical won't care and        shouldn't try to engineer that part.    -   3. Finally, the hub should record all data being received from        sensors at all times initially. You will want to know that        everything is working in the first deployed systems. In        particular, you can continue to bill for a live service even if        the subscribed (e.g. Google Nest) is not using every event        object you send. If they subscribe for a 24-7 service, then you        must provide it. Meanwhile, the data can be stripped off the hub        in files and posted back to your cloud server. This will be        invaluable to debug and cross check your initial deployed        systems, particularly if there is a billing interface involved        somewhere.

You will have a recorded “audit trail” proving that your event objectswere continuously produced and were accurate—this will be key tosustaining the value chain.

This might useful for a higher level service that does need to beworking all of the time, or all year round.

For example, a winter only service is an on-demand service that isswitched on only when the external temperature drops below a certainlevel.

At that time, the service is added into the queueing system.

Each event while having a time stamp, and while being generated on aregular enough basis, will be sent asynchronously to the subscribingdevice.

Therefore the best engineering and commercial solution is to providedifferentiation of service level, with some guarantees on maximumlatency that will be experienced. This maximum latency is engineered bydropping events from the best effort queue.

e) Apical's Software Defined Network Solution: An Event Object Switch

The Apical hub software effectively becomes an event switch withpredictable behaviour, implemented purely in software.

Vendors implementing software defined network switches for the home hubmarket could therefore take Apical's design and implement the queueingsystem in hardware.

f) Layering A Control Plane Service on top of Event Services

A further level of service can be provided by Apical to layer on top ofthe services described above.

The above services are envisaged to be based on browser based userinterfaces, which accept the events as JSON formatted objects.

The browsers or apps can use the event objects to help communicateeffectively with the user by superimposing event data onto real videostreams for example.

Another effective use might be to indicate the event inside a model ofthe home environment that the higher level service has somehowgenerated.

Apical do not need to know what the event is used for.

Apical simply guarantees a certain service level based on subscription.

A new class of service can be provided for events that will be used inreal-time control systems.

In this class of systems, the events are not being used to drive userinterfaces or communications systems.

Instead this class of systems uses the events to control other machinesto do something.

This class of events can be given its own class of service with moresynchronous type behaviour.

For example Apical might offer a regular heartbeat type delivery, whichis achieved by using a scheduler to guarantee a regular delivery ofthose events that are used in control systems.

The Apical hub software would add events from this queue to the outboundstream at regular intervals, trading off the best effort queue in orderto do this.

All of the Apical hub software switching system can be designed based onpacket switch technology with the difference that the packet is now aJSON object queued in a FIFO manner.

g) A JSON Event Object Switch

Effectively Apical will have a service aware, JSON event switch—theworld's first JSON event based queueing system with a proprietary JSONevent object.

With an engineered synchronous type output of JSON event objects, otherengineering teams could convincingly construct closed loop controlsystems for the home.

For example, a heating system will have certain inertia to change thehouse temperature and will require regular inputs to drive a PID basedcontrol loop for temperature control.

It will require a synchronous pulse of events in order to engineer areliable control loop.

This is the control plane service level that Apical hub software mightoffer for subscription.

The subscriber must specify the pulse rate and pays for a higher pulserate depending on what is on offer from the Apical queueing system.

The control plane service offering has its own queue but splits thereserved bandwidth between itself and premium services. Both do this atthe expense of the best effort services.

In a scenario where the switching bandwidth of the Apical hub softwareis reaching say 50% of its maximum, it might be advisable to instantiateeither a second instance of the Apical hub software if the channelbandwidth can handle it.

Alternatively, the local area wireless network bandwidth will have to beincreased.

Even if the channel bandwidth is increased, the Apical hub softwareswitch may still reach a processing limit and may need to be duplicated.

In this instance, subscribers are simply allocated one of many Apicalhub software switches from which their service will be supplied.

This could be a great way to scale the system in the future by allowinga virtual division of services and Apical can decide inside itsmanagement system which soft switches will handle which groups ofservices.

This would be transparent to the user of the service, but would have tobe flagged to the manager of the hub hardware device that is hosting theApical hub software switch.

h) The JSON Event Object Definition

The JSON event objects could contain the following values:

Event stats array:[unique event id, time event generated, number ofsensors based on, array of timestamps of raw data generated per sensor,time spent in queue, window length used for calculation, class ofservice value, number of subscriptions, array of subscription ids]

Event type 2D array: [presence, position, gesture, posture, movement,mood]×[different people unique ids]−values are placed in each element ofthe array which in turn could be objects or arrays themselves.

For example, the position element of the array could be co-ordinates,unique room ids, or relative positions to specially chosen objects suchas a cooker, a fridge, a remote control for the TV etc.

Personal avatar information 2D array: [estimated unique id, height,weight, sex]x[different people in the home]

Note: the unique person ID can be generated by Apical with no need toknow who they are, their name etc—that can be done elsewhere in a securehigher level system if required, e.g. Apical pass on the best face viewand a unique id is returned.

This unique id is then added to the event generated and the higher levelsystem uses it to attach a name or a face to it in order to build a morepersonal user interface on a browser for example.

i) HTML5 Ready Interface Using JSON

If each event is created as such a JSON object, it can be used directlyin building a HTML5 browser interface directly using Javascript.

This will allow a faster time to market, more rapid prototyping, as itwill be natively recognisable in the browser without any intermediatetranslation layer.

It is also a more practical and broader way to present the userinterface across both mobile, home and computer devices in adherence tobrowser standards rather than closed ecosystems such as purely an iPhoneapp.

j) Apical ART Management System

On top of all of the above, the Apical system must also have its ownmanagement and diagnostics plane of activity going on.

This will require a small reserved bandwidth channel, which can beimplemented in the Apical hub software as a small reserved bandwidthwith a unique class of service recognisable in the scheduling system.

This channel is used for a wide variety of management functions.

The first function is to send subscriber information to the hub softwarein order to create a new set of queues for that subscription.

k) Billing & Subscription Model

The subscription will have to inform the Apical hub software of avariety of elements of information and so another JSON object might bethe right choice again.

For example, the JSON object could contain the unique id of the servicerequired, the id of the subscriber, the level of service subscribed for,the rate at which events will be expected to arrive or be updated andperhaps a confidence level threshold for events.

Another interesting model is to have the subscriber indicate or statewhich sensors the subscriber has permission to access.

Imagine a situation whereby certain higher level services rely on amultitude of sensors types from different manufacturers.

Each sensor purchase and each higher level service purchase has anassociated license for use of particular sensors.

The Apical hub software could be smart enough to differentiate sensorinputs to the event calculator based on licenses indicated under a newsubscriber JSON object.

1) Data Audit Trail

The next management function is to regularly pull off a file of completedata to clear the internal buffers.

For example, every 5 minutes a file is sent to the Apical cloud to storebackup data for legal and billing reconciliation at some future time.

This clears the local buffers making it manageable in terms of memoryrequirements and storage.

m) In-Field Upgrades

The next management function is to do software upgrades live to thesystem.

This could also be the method for upgrading the software on the sensorsthemselves, by passing through messages to the sensor network using themanagement communications channel reserved in the queueing system.

n) Error & Warning Monitoring and Masking

The next function is to pass monitoring point statistics to a managementmodule either situated in the cloud software or the hub software.

These monitoring points are then passed through a mask to select themost important warnings and errors and pass those back up to the cloudsystem.

Warnings and errors may also require the generation of a warning or flagto the surrounding higher level systems in place.

Finally, billing information must have confirmation that the subscriberis actively receiving the service they asked for and at the rightservice level.

The lower down in the system, e.g. the closer to the sensors that thisis performed, the more manageable it will be at larger and larger scalesof homes and sensors within the home.

The last thing you want is to have to continuously go up an down to thecloud to perform all error and warning masking and interpretation forthe various management functions.

There should also be a secondary backup management interface into thesoftware using a local console, a command line interface or similar sothat diagnostics can be done locally or through a homeowner's mobilephone network should the WiFi and internet connection be down.

Also support engineers will need a way to get into the software incertain support scenarios where debugging is necessary but cannot beperformed due to a lack of internet connectivity,

6. Market Research and Applications

A smart network may allow Smart Devices with Spirit to be controlledthrough the ecosystem.

Market Research

Various market research are possible, such as which TV programs arebeing watched, what food is being eaten. In addition, the system mayinclude sound and voice recognition. One approach is to analyse simplethings in a camera, and analyse complex things in the hub.

For the data structure, there is an interface to the outside world. Theinterface can be an API, or a query-able database. As an example, aninterface may generate an alert that someone has gazed at an airconditioner for more than two seconds. Other devices may respond to thisalert. A tracking record may only be sent from a camera if a criterionor criteria is met.

Medical Application

Apical's system captures the person's movement in real time and that isthen used by medical analytics software, programmed for example withmedical information on muscle, bone and tissue structures, to givefeedback to the patient's doctor so that the doctor can rapidly andobjectively understand how the patient moves; this is important for newdrug trials that may affect motor performance (e.g. arthritis drugs) orfor patients in physiotherapy and also to personalise the patient'srecovery experience.

7. Use Case Examples

ART can enable multiple propositions and has been segmented in thedomestic arena into the following areas:

-   -   ART within the home:        -   ART Control        -   ART HEMS        -   ART Care    -   ART Security around the home.

ART is setup to anticipate and respond to personal requirementsdepending on the homeowner specific needs. ART will quickly become theindispensible faithful companion, which places the home individuals incontrol of the environment and knows the information required to makelife run smoothly. ART fully personalises the SmartHome through anetwork of smart device. ART is a scalable platform for the user-centricSmartHome, built on high-performance Spirit computer vision at the edge.

By using the metadata extracted from the Spirit engine, ART builds avirtualized digital representation of each individual in the home and isable to understand a wide range of behaviours, such as, but not limitedtoo:

-   -   counting the number of people in the room,    -   understanding people's pose,    -   identifying persons using facial recognition data,    -   determining where people are moving from/to,    -   extracting specific gestures by an identified individual.

Some examples of benefit of ART are listed, with use case examples:

-   -   ART will anticipate needs as you move around the home:

The future ART smart home will self learn to anticipate your needs. Asyou move around the house, ART will anticipate before you enter rooms;the lighting will adjust to preferred levels, your desired audio visualsettings will be awaiting, the room will already be at the requiredtemperature. Other smart devices in the room will have been aware ofyour imminent arrival and will have prepared accordingly, whether thisbe the coffee machine having warmed up, the room's blinds adjusted oryour favourite TV channel running.

ART's network of smART sensors around your home, anticipate your needs(and are controlled ultimately by and through you) and interact with theplethora of smart devices around your home, ensures that your futurehome is as smart and automated as you would like it to be.

-   -   “Your alarm goes off, and you start your day, a typical mid week        work-day. ART has recognied that you have started your day as        planned (ART knew this was the intention, the alarm was set the        night bfore). ART has programmed your bedroom's/bathroom's        heating around your alarm call and now anticipates that it will        be 15 mins until you arrive in the kitchen, and prepares for        your imminent arrival by switching on your coffee. Other smart        devices TV or CNN), lights, blinds, XXX, XXX are not turned on        until ART knows you are going to enter the room—ART can adapt to        even the smallest of changes in your routine, for example,        helping change—youryoungest son's diaper, hence delaying your        usual breakfast time by 10 mins.”    -   “All this automation not only provides significant ease of        living (all adjusted in real time), but through the most optimum        energy usage with save material costs in runningyour home”.

ART will be a helpful and invaluable virtual assistant, anticipatingyour needs and responding to them. ART's level of assistance iscontrolled by you, it will help you as much as you wish it to do.

-   -   ART will help you remain independent for longer:

The future ART smart home will provide increased independence for theaging population. ART will enable you to control your adapted property'ssmart devices and ensure that they are all work in concert with eachother through you. ART will understand your normal daily routines, yourbehaviour and your physical wellbeing. ART can be customized to yourspecific requirements whether that be gesture control, automatedfunctionality for you (but not your partner or carer) or that you useART to know simply that you are OK and following your typical dailyroutine and to ask ART to keep your friends and family informed of this.All this is done without intrusive video capture that has the outsideworld looking in on your daily life in the current Goldfish ‘esque’ wayof many current technical solutions.

The ART ecosystem will interact with all Smart Devices, including personfitness and health devices, where these devices will complement theknowledge of your physical wellbeing to enable a full uncompromisedunderstanding of your current situation.

-   -   “On waking up ART will sense that you are up and about, going        about your tical daily routine. Your Smart Home will respond the        your whereabouts; your wheel chair friendly doors open ahead of        time (ART anticipates your typical home movements) and your        kitchen appliances, whether this be the oven or coffee machine        will already be cooking your meal or brewingyour daily cup of        coffee. ART will also know whetheryou are alone at home 24/7,        and be able to inform anyone at home or nearby of any issues.        ART will be able to incorporate the additional vitals you        measure, whether this is blood pressure, respiratory rate. ART        will be able to inform you of the most accurate summary of your        physical wellbeing; your vitals, as well as accurate physical        movement; the number of steps you have taken without the aid of        your walker, and whether this is an improvement on the previous        days . . . it is, well done. At the end of the day, afteryou        have retired, ART can inform your nearest and dearest that all        is fine—no problems.”

ART's ability to control your smart home, and to keep you, your friendsand family informed of your health and Wellbeing is entirely controlledby you.

ART's network of smART sensors around your home are able to recognizeand anticipate your requirements and understand specific variationsdaily routines, which includes your physical and mental wellbeing, ARTincorporates your other Smart data points into its understanding. ARTcan understand when you might have fallen, when you are confused, whenyou might need to have assistance, ART also understands when all is OK,how your day has been and can share with your family and moreimportantly you this information.

-   -   ART will help you keep your children safe:

The future ART smart home will help provide you with the piece of mindof the safety of your family.

ART's network of smART sensors around your home enables you to have 24/7comfort on ensuring the safety and wellbeing for your family. ART'secosystem is controlled ultimately by and through you and ensures theplethora of smart devices around your home given you the degree ofcomfort you seek as your family grows up.

In these examples, ART could be considered a guardian angel, keeping awatchful eye over your shoulder and helping to keep your loved onessafe, of course not removing the responsibility as a parent, but helpingyou in the time of need.

There are a large number of use case examples. For example, ART may alsohelp you keep look after your property when you are away or may alsoenable your kids to learn and have fun through social interaction.

FIGS. 60 and 61 show examples of image captures of a kitchen environmentin which one or more ART sensor has been placed. A sequence of videoframes generated by the sensor is analysed and a stream of informationis generated continuously and in real-time. The digital representationsof people may be extracted on an event-triggered basis. For example, twopeople are first detected inside the kitchen. The second person isidentified as Wendy and her trajectory is extracted as ‘approaching thesensor’, the first person is identified as being George. Wendy is thenrecorded as moving away from the sensor. A third person and a fourth oneare then detected as entering the room. Finally, the fourth person isidentified as Alastair, and the following gesture: ‘Alastair is showinghis right hand’ is extracted.

FIG. 62 illustrates a comparison of the typical information preservedbetween a standard video frame and an ART video frame. With the ARTsystem, only selected region of interests (Rol) have been preserveddynamically within each frame, whereas the less useful information suchas the background has been de-pixelated. The upload capability can bemanaged dynamically by varying how far the rest of the image isde-pixelated. In this example, rectangles around each individual havebeen preserved while the rest has been de-pixelated.

FIG. 63 shows a smart doorbell system enabled by ART in which an ARTenabled sensor is placed on or near an entrance door of a home. Thesystem is able to detect an object approaching in real-time. The systemis self-learning and therefore does not require an initial calibrationsetup. Hence, the system is able to learn for example about backgroundtraffic without supervision. Unimportant information such as backgroundtraffic or ambient motion may not be reported and may be ignored.Regular behaviour, such as passing people and cars, is learnt and thenidentified as background. Any approach to the door is detected andflagged. Each visitor is identified and a push-notification is sent tothe user's mobile device. FIG. 64 shows that as someone is approachingthe home, the system may detect the face of the person, and crop thecaptured frame to extract a thumbnail of the detected face. The systemcan take a series of thumbnails and compare it to its known databaselibrary.

FIG. 65 shows an example of an even generated once the system hasrecognised the person approaching. The system may alert the owner viatext with an attached thumbnail. It may also for example send a messagethat a postman has arrived and delivered a package.

FIG. 66 shows an example of the different steps of setting up an ARTdoorbell system. After installation, the system is configured with thefamily IDs. A 30-day self-learning mode follows in which the systemlearns the IDs and the environment. The system is then setup and may becontrolled by the homeowner (manage visitor ‘circles’, daily/weeklyreports, out of the ordinary alerts). The system may also be furtherenhanced depending on the services that are needed for a particular homeusage (add further ART sophistication, elderly care, multi-room Kx,entertainment controls).

The load can also be further reduced using the ART system by onlypreserving critical details, such as only the areas of the person'sface. ART is able to associate a region of interest with a particularID. Thus ART can also learn that a particular homeowner does not need topreserve certain information. For example, a picture of a dog might notneed to be preserved and taken everyday. ART is therefore able to offera large array of dynamic management, which further reduces the load ofthe service provider local network. This will be crucial as the numberof homes that the service provider is managing increases.

ART is able to count people, identify them and also differentiate thembetween known and unknowns. Only certain people are able to control thesensor with pre-programmed specific gesture(s).

In summary, ART places the homeowner at the heart of the SmartHome andplays the role of the “faithful friend” that thinks of everything. Italso implements unique methods of introducing Avatars in the home.

Examples of features available are, but not limited to:

-   -   ART recognizes you and greets you.    -   ART knows your family.    -   ART warns you of danger.    -   ART recognizes a fall, stall or a crawl.    -   ART knows when you are happy.    -   ART responds to your gestures.    -   ART spots strangers.    -   ART recalls and responds to your habits.    -   ART builds useful data for analysis.    -   ART watches over your pets.

ART also provides the means to optimize energy management, act as anintelligent security control and be the activation interest for allsmart home devices. Some of the benefits or advantages for differentplayers may include, but not limited to:

-   -   Utility provider:        -   brand enhancement,        -   proactive guidance,        -   transition to value added relationship service provision—ART            Landlord.    -   Green deal-up sell:        -   accredited CAP rating,        -   expanded landlord services,        -   optimised energy efficient homes.    -   Residential landlords:        -   absent periods- adherence,        -   monitoring pet policy,        -   understanding number of people living at property,        -   confirmation of areas off-limits (balconies, swimming pool),        -   specific tenant behaviour—electricity usage, water usage            (with smart meter),            -   green tenant ‘awards’,        -   EPC accreditation.    -   Commercial landlords:        -   liability management—number of people with the premises,        -   fire drills H & S—confirmation on evacuation,        -   lift systems—adherence to max persons.

FIG. 67 shows an example of care monitoring supported by ART. ART Careenables an understanding of daily movements, such as ‘Gail gets uparound 8 am’ (1), ‘Gail has her carer visit at 10 am—stays for 15 mins’(2 and 3), ‘Gail leaves the house with her carer’ (4), ‘Gail returns at2 pm’ (5), ‘Gail watches TV for 3 hours in the afternoon’ (6). Alertscan be setup and arranged to ensure that specific movement orinformation are known, such as ‘Gail not left bedroom by 9:30 am’,‘Carer doesn't visit’, ‘Carer only stays 3 mins’, ‘Gail does returnhome’ or ‘Gail leaves the house at 11:08 pm’.

FIG. 68 shows an ART system demonstration of a real time scenario takingplace inside the home. The demonstration setup does not use any cloudbased systems. Therefore all of the component parts for thedemonstration can be brought into a briefing room with no reliance onInternet access or external server software to enable the demonstrationto take place. The basic components are an Apical enabled camera sensor,the Apical Hub “Heart” software and a controllable electrical devicesuch as a light switch. The existing Apical server may be used to hostthe Apical Hub Heart software. A camera with direct feed into the servercan be used as the sensor for the home. The output of the Apical HubHeard software is transmitted to the demo device controller. Thiscontroller may not physically be located in the Apical Server, and itmay be located anywhere as long as it is connected to the Apical HubHeart software over a local network of some description. Apical HubHeart software is thus capable of sitting in any 3^(rd) party device inthe home which provides implementation flexibility for the customer.However it does not have to reside in the controllable device or in itscontroller software. This ensures that multiple 3^(rd) party controlsystems can use the Apical Hub Heartbeat that the software produces.

Demonstration walkthrough in real-time: first, the mobile device such assmart phone may take a picture of the two executives inside the room.The purpose of this photo shot is to enable the Apical system torecognise both executives and to assign control capability to only oneof the executives for the purposes of demonstrating “ID” controlcombined with “Gesture”. The control settings on the demo devicecontroller are adjusted to reflect the permissions granted to oneexecutive and not to the other. The camera is then pointed at theexecutives and they are asked to make a gesture in turn, one after theother. This is fed into the Apical Hub Heart software, which creates twoevents. The first event is the executive without permission to controlthe device as they make their gesture. The second event is the executivewho has permission to control the device as they make their gesture. Thestream of raw sensor data from the camera is converted inside the serverinto an XML stream using the Apical FPGA and embedded code. This XML issent to the Apical Hub Heart software—a separate application that may berunning on the same server. This software could be running on any devicein the home. The Apical Hub Heart software creates an event from the XMLdata for the “Gesture” and “ID” services and pushes them out as JSONobjects to the smart device (phone, tablet or laptop) that iscontrolling the home device (e.g. a light or a fan etc). The HeartbeatEvents are sent to the device controller, which can cross check the IDof each event that it receives. It then checks its local rules that weresetup at the start of the demo, and only turns on the device when anevent is received from the correct executive to do so. The result isthat the light or device is only turned on when one of the customerexecutives in the room makes a gesture and it is not turned on when theother customer executive makes the same gesture. This clearlydemonstrates the ID and Gesture event generation working in tandem toallow smart home control.

There are three main components to the demo in terms of devices andsoftware that need to be created/assembled:

-   -   1. The existing Apical server with a camera attached,        functioning to be able to perform gesture recognition and which        may also use 3^(rd) party software for facial recognition all in        real time.    -   2. The Apical Hub Heartbeat software that can receive the XML        stream from the FPGA and convert that into events in JSON        format, pushing them out on a network port to the final device.    -   3. A smart phone, tablet or laptop device which is networked to        the above server. It contains some control software that        controls a physical electrical device that is attached to it        using a Digital-to-Analog Convertor card, a power source, a        switch and a light bulb for example. The controller app must be        able to read the event's gesture and ID arriving in JSON format,        and hold a set of rules to allow or disallow the gesture        depending on the ID. The controller must then act on the gesture        control and turn the light on or off depending on the current        state of the switch. A combination of a laptop and a smart phone        might work here with a DAC card attached to the laptop using the        USB port. Note the DAC could be a Raspberry Pi control board for        example for demonstration purposes.

This setup may be any off-the-shelf lighting control kit that uses asmartphone or laptop based control app. The Software to receive the JSONevent and interface into the controller may be created. This may also beimplemented using a laptop, a DAC card controlling a power switch—a lampor any electrical device could be attached. The server must communicateover a wired Ethernet cable or Wi-Fi to the lighting controller—it sendsa JSON Event object containing the gesture control information. Thecurrent server in this demonstration comprises a camera attached feedinginto a FPGA analysis.

FIG. 69 illustrates an implementation of ART inside a vehicle. Arear-facing camera may be positioned on or in relation to the vehiclerear view mirror such that it captures passengers and the driver. TheART system may monitor the inside of the vehicle in real time providingcare and protection of all the passengers (including the driver).

The setup as shown in FIG. 69 may also be used in a transportationvehicle for hire such as a taxi. The ART system may then provide a widerange of useful information about the taxi driver as well as the taxipassengers.

The data extracted by the ART system may be integrated directly with thetaxi company existing mobile or web application. This is illustrated inFIG. 70, a customer may hail a taxi via a mobile application (1), andthe camera placed around the rear view mirror may capture a video, animage or thumbnail of the passenger (2). The image/video or thumbnailand/or additional data may be stored remotely from the system, such asin a secure cloud (3). It may be stored for a limited time (for example30 days). An enhanced mobile user experience may be provided (4) whereinthe mobile application is updated with an image captured previously. Theimage may also be refreshed on the mobile application automatically orat specific time intervals. Hence, the taxi company ‘knows’ thepassenger is on board. In addition, the customer also knows that thetaxi company knows and that the ART system knows.

The image or video upload to a remote location such as in a secure cloudis managed through Spirit, in which 3G and 4G uploads are supported(frame by frame). All passengers (including the driver) may be monitoredin real-time, and the number of people inside a vehicle may be known atall times. The customer may opt in the facial identification and maytherefore register in real time.

The journey's time, images and ride information may also be shared withothers, such as a parent or guardian who wishes to access the well beingof the passenger. A parent may therefore get notified for example whenthe passenger is safely on board and of the estimated time of arrival.

Detailed knowledge of driver behaviour may be gathered, such aslocation, how long the driver has been driving, how rapid or measuredthe car's acceleration is, how harsh or smooth the car's braking is, howhard or gentle the car's cornering. From this data, detailed profile ofdriver's behaviour may be built, making it possible to, for example:

-   -   Rate the driver,    -   Include acceptable behaviour in T&Cs,    -   Develop star club—points for best drivers,    -   Wear your ‘star with pride’.

This application has several benefits for the rider, such as peace ofmind, detailed drivers history available, retail record of journey orbackup history if needed. The driver may also benefit from for example,enhanced history, lower insurance costs, satisfied riders and fewerdeclined riders.

In addition, a forward facing camera may be utilised that understandsthe road conditions and environment such as proximity of other cars andpedestrians and other vital information in case of a collision.

Further examples of use cases for ART, AWARE and ALIVE are provided inthe following tables.

ART use case examples:

TABLE 3 Security around the home (Figures 63-65) Use Case TitleDescription Smart door bell: Self An important aspect of the inventionis that the system is self-learning. learning system Hence ART does notneed a calibration at setup. In time, it will be able to identify normalpathways, and the accurate location of a footpath leading to an entranceof a door. The system is also able to recognise that people are walkingup and down a footpath. The system is able to detect when someone isapproaching a particular home in real time. The system is also able tolearn what is unimportant information and learns to ignore it. Forexample, the system learns that ambient movement is unimportant byidentifying that movement never comes up to a door. The system alsolearns that flapping leaves in the background are not important, byidentifying it as not being human activity. The engine is also trainedaround cars and is able to learn that cars are moving up and down theroad. Smart door bell: The system is able to compare a known to a knownand a known to a Best capture unknown. It can take a series ofthumbnails and compare them to its thumbnail from a known library, anddo a number of interactions depending on whether live video footagethere is a known or unknown individual outside the house. For example,and compare it in it can alert the owner via text with an attachedthumbnail if there is real-time someone strange outside, or send amessage that a postman has arrived and delivered a package. Detectingunusual Apical is currently working on being able to understandrobberies. For events: understand example, the system understands thefact that it is a human taking robberies something from a car, despitethe human being an atypical shape by: having a hoody and hat on,skulking across an unknown pathway crunched over. SmART Delivery When adelivery is expected from a postman, the delivery service sends athumbnail of the employee delivering the package to the system. Thethumbnail is linked to the library, which is linked to the smartdoor.The system is able to extend the “known” inside the library temporarily,next the system compares what is visible on camera to its library, andif there is a match, the delivery box is opened. Room occupancy ART candetect room occupancy and detect abnormal conditions such as unsafeones. For example: how many people are inside a car, or inside a lift,or inside a night clubρ? Is the safe limit exceeded?

TABLE 4 Control Use Case Title Description Self learning Once you have anetwork of sensors around the home, there is no need to tell the systemsystem “that's a doorway”, as it will learn known pathways through thehouse, and it will identify people and for example will learn how theytypically move, allowing a lot of predictability. The system is able tobuild up distances, furniture position, window position, and roomconnectivity on its own with no user inputs. The system can obtain anaccurate depth of field map, based for example on a known human averageheight. The system is able to know/learn the size of a room. The systemis able to quickly learn doorway(s) and footpath(s). It also learnstypical behaviours, such as when children are coming back from schoolfor example. Understanding ART smart home will self learn to anticipateyour needs. It is based on a network of room of sensors around the home,which understand room occupancy, and numbers of occupancy individuals,allowing individual temperature preferences (ex: He/She prefers the androom to be 18 as opposed to 22-sensors connected to thermometer). ART'santicipating network of sensors interacts with the plethora of smartdevices around the home. individual ART anticipates needs as people movearound the home and responds to them. needs For example, ART anticipatesbefore a person enters a room and adjusts lighting to preferred levelswhile the desired audio visual settings awaits and the room is alreadyat the required temperature. Other smart devices in the room are awareof imminent arrival and have prepared accordingly, whether this be thecoffee machine having warmed up, the room's blinds adjusted or afavourite TV channel running. ART system understands preferences for aweekday morning (which may vary depending on the day/time) and the smARThome responds accordingly. For example: “Your alarm goes off, and youstart your day, a typical mid week work-day. ART has recognized that youhave started your day as planned (ART knew this was the intention, thealarm was set the night before). ART has programmed yourbedroom's/bathroom's heating around your alarm call and now anticipatesthat it will be 15 mins until you arrive in the kitchen, and preparesfor your imminent arrival by switching on your coffee. Other smartdevices TV (for CNN), lights, blinds, XXX, XXX are not turned on untilART knows you are going to enter the room. ART can adapt to even thesmallest of changes in your routine, for example, helping change youryoungest son's diaper, hence delaying your usual breakfast time by 10minutes. All this automation not only provides significant ease ofliving (all adjusted in real time), but also provides the most optimumenergy usage. Smart An ART enabled sensor will recognize a person hasentered a room [avoiding your Lighting family dog triggering aresponse]. Art ID [optional functionality] will set the smart lightingto your personal preference. ART enabled sensor will recognize pre-setgesture/identity combinations to allow you to control your smartlighting (different tones, on/off, auto switching all lights off when noone is home). When ART is able to help you to look after your propertywhen you are away individuals (Party - Facebook ... avoid the housetrash). are away ART links ART links with schedule and outlines anynecessary information pertinent to one's with each day's schedule (andthat of one's family). ART may remind that one of the childrenindividual's has a dentist appointment, and whether a child has awokenyet. The ART system is schedule able to recognize individuals within ahome setup before they walk into the kitchen first thing in the morning,ART (in connection with smart speakers) informs of the weather andcommute update for the day and provide options if there are changes tothe usual route. If individuals within a home setup drive to work andparked their car on the street, ART is able to remind anyone where thecar was left the night before.

TABLE 5 HEMS Use Case Title Description Energy monitoring Measurement ofthe temperature as a function of time when people enter/leaves.Calculation of individual carbon footprint contributions - leavinglights and other devices on, heating mis-use etc. Energy Children (andother family members!) continually leave lights, heating andsaving/climate A/C, TVs, play-stations et al on throughout thehouse-devices and control appliances automatically turned off (andanticipated when need to be on) when they are required. ART enabledtemperature/occupancy sensors provide real time data, 24/7 to allow youto optimize your home's heating and energy controls, hence maximizingyour energy savings. “Heating friend” ART will optimize based on actualreal-life requirements the energy needs of the home [rather than apre-programmed, manually influenced inputs]. If the system is connectedto your heating system, it can quickly understand the energy efficiencyof the room. For example, if heating is off inside a room and no one isoccupying the room, it is possible to understand how quickly the roomloses temperature. This can change the relationship between the productand the consumer, becoming almost like a “heating friend”, giving adviceon how money can be saved by changing energy consumption. The governmentcould use it to manage climate control. This also changes the game for autility provider. (Example: you can offer an install of the latestdouble glazing because you know how much it is going to save and affordit at no cost because you understand the savings). Real time EPC As of2018, a new regulation in England and Wales will require that ratingproperties rented out have a minimum performance rating of E on anEnergy Performance Certificate (EPC). The system measures real time EPCrating that could be automatically validated to the current EPC rules.

TABLE 6 Education Use Case Title Description Education tool ART enableskids to learn and have fun through social interaction.

TABLE 7 Care Use Case Title Description Understand ART is able to learnnormal behaviour. Understand if a family member has gotten normalbehaviour up and if they are going to the kitchen according to theirnormal routine for the day. from unusual ART understands if someone doessomething and keeps doing it again a few behaviour instants later(Alzheimer's case). For example, do they spend an unusual amount of timestaring at the window? Are they constantly going back to the window? Arethey doing something, and 5 minutes later are they doing it again?Maintaining ART family alert gives your family and friends piece ofmind, 24/7 that you independence are OK. ART family alert recognizesyour daily routines, that all is normal. without the need Importantly,ARTfamily knows immediately if there is a problem. ART Care forsmartphones can help the elderly byknowing if an individual has fallen,if they remain on and wearables./ the ground, or if they got upagain, ifthey are still for an unusual period of time Detect falls and if theyarefollowing their regular routine using sensors. ART Care can furtheralert a carer or family member in close proximity. Care + Smart doorART's doorbell can make sure the correct carer has visited and whetherthey bell are on time. Mood assessment From captured thumbnails, it ispossible to extract and/or assess the mood of a person. Baby monitoringand wellbeing: the system may extract accurate facial capture foremotion monitoring. NHS Monitoring of individuals (i.e elderly) andmetabolism calculations “Guardian angel” TheART smart home help providesyou with the piece of mind of the safety of your family. ART's networkof smARTsensors around your home enables you to have 24/7 comfort onensuring the safety and wellbeing for your family. ART's ecosystem iscontrolled ultimately by and through you and ensures the plethora ofsmart devices around your home given you the degree of comfort you seekas you family grows up ART identifies/warns of children at risk. When achild comes home, tracking might be enabled (ROI encoding) - for examplea child is approaching the hot oven, when Parent or Child-minderinadvertently distracted elsewhere. For example, “ART's sensors would beable identify whether your toddler, George, is crawling towards thestairs and your stair-gate has inadvertently been left open. ART wouldgive a verbal warning over your home's connected speakers, in the actualroom you (or others) are in, as ART would know where you are in the homeat that time.” “In a similar way, ART's kitchen sensor will be able tosense if your eldest, a 4 year old called Philip, was walking towardsthe cooker whilst it was on with pans of boiling water, you might be atthe other end of the dinning area and not in the immediate vicinity. ARTwould again give you a verbal warning over your home's connectedspeakers”

TABLE 8 AWARE, Retail Use Case Title Description Understanding Spiritengine in the retail context will enable to understand how shoppersbehave in a of people retail environment. Information extracted inreal-time include: people count, dwelling behaviour time, gaze time.Interaction with Example of an Interactive display with a shopadvertising. AWARE understands in the customer by real-time if aperson's attention has been grabbed and for how long. AWARE isresponding to able to change or respond to a person's behaviour inreal-time through the ability their behaviour, to interact with them. Asan example, if someone is excited about a particular ad and by trying towith a specific product, AWARE can track them as they go towards theproduct change their (or start going away from it), and learns how toenhance their shopping behaviour, experience. Store Shoppers flow can beanalysed in order to understand the effectiveness of a management shoplayout. AWARE is also able to understand and monitor the effectivenessof the shop management/workers. Heat maps AWARE is able to produceheatmaps describing where people have been and for how long. Theheatmaps can be overlaid with a scene of the space and create a 3D mapof how the space has been used by people during periods of time. Thisenables to maximise profit per square foot by, for example learning onhow to direct people to right areas with the right products. One to onereal- AWARE provides the ability to interact on a one to one basis withindividuals. time marketing For example, a known shopper that has signedup and gave the ability to use their facial feature, walks past a windowdisplay and spends a certain amount of time interacting there. Dependingon how long they have spent looking at the advert, they might be given adiscount for purchasing the product on that particular day. One to onereal-time marketing can be directed towards short- life product in orderto maximise the obtainable value for the stock on that day, such thatproducts are not thrown away. One to one real- One to one marketing isachieved by detecting a customer from tracking time marketing numberplates and by learning of the customers buying habits, and sending +License plate them the most appealing deal possible by text for exampleas they wait for their recognition car to fill up. AWARE may detect aplate number and may also learn who the person/shopper is driving thecar associated to the detected plate number. AWARE may learn whichindividual within a household is the most susceptible to drive the carfor this particular day and time. For example, AWARE may have learntthat Mrs M. on Monday drives the family car and that she usually spends10 minute to fuel the car. Thus, it may be possible to decide to giveher a time-limited coupon to a nearby shop for that day. Real timedigital This use case relates to advertising spaces located in busstops, tube, highways public or any public spaces. AWARE can providereal time advertising through digital advertising space space based onthe number of eyes looking and people identified that regularly walkthrough or near the advertisement. Real time Heat maps are generated totrack and analyse people flow and their responses analysis of to thechange in advertisement. Therefore a huge amount of people flow beextracted in order to learn for example which advertisements aresuccessful. This can have effects all the way back to the advertdesigners, hereby completely changing how the market is currently beingoperated.

TABLE 9 Other use cases Use Case Title Description Extreme work A systemthat always knows people presence and understands individual presence.environment The system knows where people are 24/7, and can track shiftchangeovers. It knows where people should be at any point in time andchecks whether the operational protocol has been followed correctly ornot. Emergency In case of an emergency, emergency services can godirectly to where people are and services where they are trapped. DataValidation A system to provide big data validation for existence ofpeople, buildings, furniture, movement of crowds to back up auditabledata trails and validate other calculations for governments,institutions and businesses.

Note

It is to be understood that the above-referenced arrangements are onlyillustrative of the application for the principles of the presentinvention. Numerous modifications and alternative arrangements can bedevised without departing from the spirit and scope of the presentinvention. While the present invention has been shown in the drawingsand fully described above with particularity and detail in connectionwith what is presently deemed to be the most practical and preferredexample(s) of the invention, it will be apparent to those of ordinaryskill in the art that numerous modifications can be made withoutdeparting from the principles and concepts of the invention as set forthherein.

1. A driver monitoring system for monitoring a driver of a vehicle, thesystem including: (i) a forward facing camera arranged to capture animage of the external environment; (ii) a rear facing camera arranged tocapture an image of the driver; (iii) a computer vision sub-systemconnected to the cameras and programmed to track objects; (iv) a driverbehaviour sub-system connected to the computer vision sub-system andconfigured to gather information on the driving behaviour of the driver.2. The driver monitoring system of claim 1 in which the computer visionsub-system is an edge based sub-system that is located in the vehicle.3. The driver monitoring system of claim 1 in which the computer visionsub-system monitors the inside of the vehicle.
 4. The driver monitoringsystem of claim 1 in which the computer vision sub-system monitors theinside of the vehicle in real time for the purpose of providing care andprotection for the driver.
 5. The driver monitoring system of claim 1 inwhich the computer vision sub-system monitors road conditions.
 6. Thedriver monitoring system of claim 1 in which the computer visionsub-system monitors the proximity of other cars to the vehicle.
 7. Thedriver monitoring system of claim 1 in which the computer visionsub-system monitors information in case of a collision.
 8. The drivermonitoring system of claim 1 in which the computer vision sub-systemdoes not output live continuous or streaming video to an externalserver.
 9. The driver monitoring system of claim 1 in which the computervision sub-system outputs video to an external server only if apredetermined event occurs.
 10. The driver monitoring system of claim 1in which the computer vision sub-system has been trained using machinelearning.
 11. The driver monitoring system of claim 1 in which thecomputer vision sub-system applies feature extraction and classificationto find objects of known characteristics in each video frame.
 12. Thedriver monitoring system of claim 1 in which the computer visionsub-system applies deep learning feature extraction and classificationtechniques to find objects of known characteristics in each video frame.13. The driver monitoring system of claim 1 in which the computer visionsub-system monitors the inside of the vehicle in real time for thepurpose of providing care and protection for any passengers.
 14. Thedriver monitoring system of claim 1 in which the computer visionsub-system analyses of one or more of the trajectory, pose, gesture,identity of the driver or any passenger in the vehicle.
 15. The drivermonitoring system of claim 1 in which the computer vision sub-systeminfers intent of the driver or any passenger in the vehicle through ananalysis of one or more of the trajectory, pose, gesture, identity ofthe driver or any passenger.
 16. The driver monitoring system of claim 1in which the computer vision sub-system detects people by extractingindependent characteristics including one or more of the following: thehead, head & shoulders, hands and full body, each in differentorientations, to enable an individual's head orientation, shoulderorientation and full body orientation to be independently evaluated. 17.The driver monitoring system of claim 1 in which the computer visionsub-system uses data from multiple camera sensors, each capturingdifferent parts of an environment, to track and show an object movingthrough that environment and to form a global representation that is notlimited to the object when imaged from a single camera sensor.
 18. Thedriver monitoring system of claim 1 in which the driver behavioursub-system is configured to gather information on how rapid or measuredthe vehicle's acceleration is.
 19. The driver monitoring system of claim1 in which the driver behaviour sub-system is configured to gatherinformation on how harsh or smooth the vehicle's braking is.
 20. Thedriver monitoring system of claim 1 in which the driver behavioursub-system is configured to gather information on how hard or gentle thevehicle's cornering is.
 21. The driver monitoring system of claim 1 inwhich the driver behaviour sub-system is configured to enable a driverprofile or driver rating to be generated.
 22. The driver monitoringsystem of claim 1 in which the driver behaviour sub-system is configuredto gather information on how long the driver has been driving thevehicle.
 23. The driver monitoring system of claim 1 in which the driverbehaviour sub-system is configured to gather information on passengerbehaviour
 24. The driver monitoring system of claim 1 in which therear-facing camera is positioned on or in relation to the internalvehicle rear view mirror.
 25. The driver monitoring system of claim 1 inwhich the computer vision sub-system performs real-time virtualisationof a scene, generating a virtualised or digital representation thatdefines the appearance of a generalized driver or any passenger, and notthe specific driver or any passenger, in which a person is representedas one of the following: a standardised shape, a flat or 2-dimensionalshape including head, body, arms and legs, a symbolic or simplifiedrepresentation of a person, or an avatar.
 26. The driver monitoringsystem of claim 1 in which the computer vision sub-system generates adigital representation in which symbolic or simplified representationsof different people are distinguished using different colours.
 27. Thedriver monitoring system of claim 1 in which the computer visionsub-system is switchable between (i) a first mode in which it generatesa digital representation that is not a photographic image or video imageand does not enable a photographic or video image of a person to becreated from which that person can be recognised and (ii) a second modein which a photographic image or video image is generated.
 28. Thedriver monitoring system of claim 1 in which the computer visionsub-system generates a digital representation that relates to one ormore of: animals, pets, inanimate objects, dynamic or moving objects,moving vehicles.
 29. A passenger monitoring system for monitoring apassenger in a vehicle, the system including: (i) a forward facingcamera arranged to capture an image of the external environment; (ii) arear facing camera arranged to capture an image of the passenger; (iii)a computer vision sub-system connected to the cameras and programmed totrack objects; (iv) a passenger behaviour sub-system connected to thecomputer vision sub-system and configured to gather information on thebehaviour of the passenger.
 30. A method of monitoring a driver of avehicle and any passengers in the vehicle, the method including thefollowing steps: (i) generating data at a forward facing camera arrangedto capture an image of the external environment; (ii) generating data ata rear facing camera arranged to capture an image of the driver and anypassengers; (iii) sending the data from the cameras to a computer visionsub-system that is programmed to track objects in the externalenvironment; (iv) operating a behavioural analysis sub-system that isconfigured to gather information on the driving behaviour of the driverand the behaviour of any passengers, using data from the computer visionsub-system.